This notebook is a documentation of all the code that I am writing for my Year 11 Personal Project, which revolves around building a machine learning (ML) model to be able to predict the type of object present in a given image.
In this directory there is a large (2GB) dataset with images, which has a total of 256 different categories. Each category corresponds to a specific object. Soon enough, this dataset will be split into training, validation, and test sets. There will also be additional testing data consisting of pictures taken by me.
The first step required is to load the dataset.
As written above, there is a large dataset in this directory. I will need to load the dataset so that it is of a format that is more suitable for data analysis (more about this part later) and ML.
An issue that I did not previously identify is that I do not know how to load images into numpy arrays (which are much more suitable formats to work with). Numpy arrays can easily be converted into Pandas data frames for the exploratory data analysis (EDA) and can be used for many other different purposes.
As a result, I have done some research about this.
# Loading all necessary modules
import numpy as np
import pandas as pd
from matplotlib import pyplot as plt
import imageio
import utils # some utility functions that I wrote
# This just makes it easier for me later so I don't need to retype this name
DATAPATH = "./256_ObjectCategories"
It only took a few minutes before I found a useful module. The imageio module provides functionality to load an image into a np.array format, which is precisely what I needed!
I found it on this website[1] which presents tutorials using scipy. It made use of imageio as an auxiliary module. According to its PyPi page[2], imageio is quite a popular module, with 1322 stars on GitHub at the time of writing. It is also actively maintained - its latest version was released on October 2, 2023.
In the cell below, I load one of the images from the dataset into a np.array.
test_arr = imageio.imread(DATAPATH + '/001.ak47/001_0001.jpg')
# Explore some basic properties of `test_arr`
utils.output({
'Total number of elements': test_arr.size,
'Shape': test_arr.shape,
'Example element': test_arr[0][0]
})
Total number of elements: 416166 Shape: (278, 499, 3) Example element: [ 94 107 115]
From the above we can get that the image 001_0001.jpg, which is an image of an AK-47 gun, has dimensions of 278 by 499 pixels. The fact that the shape is (278, 499, 3) - note the 3 at the end - confirms that we have in fact been able to receive all three colors (red, green, blue) from the image.
Let's try another image.
other_arr = imageio.imread(DATAPATH + '/006.basketball-hoop/006_0005.jpg')
# Explore some basic properties of `other_arr`
utils.output({
'Total number of elements': other_arr.size,
'Shape': other_arr.shape,
'Example element': other_arr[0][0]
})
Total number of elements: 54960 Shape: (229, 240) Example element: 0
This is problematic.
The two main problems that I can identify are:
From this I can conclude that my selected dataset is not great for ML purposes.
After a bit of hunting around, I found the CIFAR-100 dataset.[1] This dataset contains 60000 32x32 images. The low resolution is slightly disappointing because I was hoping to be able to use this model at the exhibition and allow people to give it images, but I suppose it is a necessary evil to ensure that training does not take too long. This dataset is suitable because there are 100 categories organized into 20 higher-level "superclasses".
[1] https://www.kaggle.com/datasets/fedesoriano/cifar100/data
Let us load the dataset. Its format is different from the other one. I am using a method described on the kaggle page. (kaggle is a website where people can host datasets for data analysis and ML.)
import pickle
with open('train', 'rb') as file:
dataset = pickle.load(file, encoding='bytes')
all_data = dataset[b'data']
# Explore some basic properties of `all_data`
utils.output({
'Total number of elements': all_data.size,
'Shape': all_data.shape,
'Example element': all_data[0]
})
Total number of elements: 153600000 Shape: (50000, 3072) Example element: [255 255 255 ... 10 59 79]
It is clear that this dataset is organized in a different manner from the other dataset (the one with 256 categories).
One key thing to note is that they have split the dataset into a training set and a test set in a 5:1 split. My machine learning work method requires a validation set.
I have decided on the following distribution:
In order to ensure that all classes are represented across all sets, I need to write some code to distribute them.
# Step 1: Transform the objects' dimensions.
all_data = all_data.reshape([50000, 3, 32, 32])
Before proceeding, let us check the images to see whether they are up to our standards for machine learning training.
plt.imshow(np.transpose(all_data[0], (1, 2, 0)))
<matplotlib.image.AxesImage at 0x2470004def0>
As I already said, that the 32x32 dimensions are problematic for us.
It's time to split the dataset into training and validation sets. Further, the shape of this dataset is quite inconvenient, so I need to reshape all of the arrays.
reshaped_data = np.zeros((50000, 32, 32, 3))
for i in range(50000):
if i % 100 == 0: print(i, end=" ")
for j in range(32):
for k in range(32):
for l in range(3):
reshaped_data[i][j][k][l] = all_data[i][l][j][k]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500 15600 15700 15800 15900 16000 16100 16200 16300 16400 16500 16600 16700 16800 16900 17000 17100 17200 17300 17400 17500 17600 17700 17800 17900 18000 18100 18200 18300 18400 18500 18600 18700 18800 18900 19000 19100 19200 19300 19400 19500 19600 19700 19800 19900 20000 20100 20200 20300 20400 20500 20600 20700 20800 20900 21000 21100 21200 21300 21400 21500 21600 21700 21800 21900 22000 22100 22200 22300 22400 22500 22600 22700 22800 22900 23000 23100 23200 23300 23400 23500 23600 23700 23800 23900 24000 24100 24200 24300 24400 24500 24600 24700 24800 24900 25000 25100 25200 25300 25400 25500 25600 25700 25800 25900 26000 26100 26200 26300 26400 26500 26600 26700 26800 26900 27000 27100 27200 27300 27400 27500 27600 27700 27800 27900 28000 28100 28200 28300 28400 28500 28600 28700 28800 28900 29000 29100 29200 29300 29400 29500 29600 29700 29800 29900 30000 30100 30200 30300 30400 30500 30600 30700 30800 30900 31000 31100 31200 31300 31400 31500 31600 31700 31800 31900 32000 32100 32200 32300 32400 32500 32600 32700 32800 32900 33000 33100 33200 33300 33400 33500 33600 33700 33800 33900 34000 34100 34200 34300 34400 34500 34600 34700 34800 34900 35000 35100 35200 35300 35400 35500 35600 35700 35800 35900 36000 36100 36200 36300 36400 36500 36600 36700 36800 36900 37000 37100 37200 37300 37400 37500 37600 37700 37800 37900 38000 38100 38200 38300 38400 38500 38600 38700 38800 38900 39000 39100 39200 39300 39400 39500 39600 39700 39800 39900 40000 40100 40200 40300 40400 40500 40600 40700 40800 40900 41000 41100 41200 41300 41400 41500 41600 41700 41800 41900 42000 42100 42200 42300 42400 42500 42600 42700 42800 42900 43000 43100 43200 43300 43400 43500 43600 43700 43800 43900 44000 44100 44200 44300 44400 44500 44600 44700 44800 44900 45000 45100 45200 45300 45400 45500 45600 45700 45800 45900 46000 46100 46200 46300 46400 46500 46600 46700 46800 46900 47000 47100 47200 47300 47400 47500 47600 47700 47800 47900 48000 48100 48200 48300 48400 48500 48600 48700 48800 48900 49000 49100 49200 49300 49400 49500 49600 49700 49800 49900
# Testing
reshaped_data[0][0]
array([[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 252., 248.],
[231., 222., 213.],
[176., 168., 163.],
[237., 236., 235.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[255., 255., 255.],
[252., 255., 255.],
[242., 252., 252.],
[229., 240., 234.],
[195., 205., 193.],
[212., 224., 204.],
[182., 194., 167.]])
I need to understand how the dataset is organized in order to properly split the dataset so that all classes are represented. Reminder: 100x400 images in training, 100x100 images in validation.
dataset[b'coarse_labels'][:10]
[11, 15, 4, 14, 1, 5, 18, 3, 10, 11]
As can be seen, the dataset is pretty disorganized, so it is necessary to populate the train and validation sets carefully so that each class is represented 400 times in training, and 100 times in validation.
This is a program to create the proper sets.
classified_data = np.zeros((20, 2500, 32, 32, 3))
__sizes = [0] * 100
for i in range(50000):
label = dataset[b'coarse_labels'][i]
classified_data[label][__sizes[label]] = reshaped_data[i]
__sizes[label] += 1
train_data = np.zeros((20, 2000, 32, 32, 3))
valid_data = np.zeros((20, 500, 32, 32, 3))
for i in range(20):
for j in range(2500):
if j < 2000: train_data[i][j] = classified_data[i][j]
else: valid_data[i][j - 2000] = classified_data[i][j]
"""
1) aquatic animals (classes 1-10)
3) flowers (classes 11-15)
4) food containers (classes 16-20)
5) fruit and vegetables (classes 21-25)
6) household electrical devices (classes 26-30)
7) household furniture (classes 31-35)
8) insects (classes 36-40)
9) large carnivores (classes 41-45)
10) large man-made outdoor things (classes 46-50) NOT INCLUDED
11) large natural outdoor scenes (classes 51-55) NOT INCLUDED
12) large omnivores and herbivores (classes 56-60)
13) small to medium sized mammals (classes 61-65, 81-85)
14) non-insect invertebrates (classes 66-70)
15) people (classes 71-75)
16) reptiles (classes 76-80)
17) trees (classes 86-90)
18) vehicles (classes 91-100)
"""
None
Above, you can see that I have made some choices about which classes to keep and I have also merged some classes together. These are based on considerations related to the feasibility of distinguishing between similar classes (e.g. small vs. medium sized mammals). I'm not sure whether large carnivores should me merged with large omnivores/herbivores but I have decided to leave them separate.
df = pd.DataFrame(columns=[*range(3073)])
df.head()
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 3063 | 3064 | 3065 | 3066 | 3067 | 3068 | 3069 | 3070 | 3071 | 3072 |
|---|
0 rows × 3073 columns
df = pd.DataFrame(columns=[*range(3073)], index=[*range(45000)])
df.head()
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 3063 | 3064 | 3065 | 3066 | 3067 | 3068 | 3069 | 3070 | 3071 | 3072 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 1 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 2 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 3 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
| 4 | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | ... | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN | NaN |
5 rows × 3073 columns
ranges = [[0, 1], [2, 2], [3, 3], [4, 4], [5, 5], [6, 6], [7, 7], [8, 8], [11, 11], [12 ,12, 16, 16], [13, 13], [14, 14], [15, 15],
[17, 17], [18, 19]]
assignments = [-1] * 20
i = 0
for r in ranges:
if len(r) == 4:
for k in [12, 16]:
assignments[k] = i
else:
for k in range(r[0], r[1] + 1):
assignments[k] = i
i += 1
'''y = 0
for i in range(100):
print(i)
for j in range(400):
x = 0
for k in range(32):
for l in range(32):
for m in range(3):
df[x][y] = train_data[i][j][k][l][m]
x += 1
df[3072][y] = assignments[i]
y += 1'''
lengths = [2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500, 2500]
classified_data_2 = np.zeros((37500, 1, 32, 32, 3), dtype=np.uint8)
y_data = np.zeros((37500))
ptrs = []
train_ptr = 0
valid_ptr = 30000
for i in range(20):
print(i, end=" ")
if assignments[i] == -1: continue
length = len(classified_data[i]) // 2 if i in [0, 1, 12, 16, 18, 19] else len(classified_data[i])
for j in range(length):
if j >= length * 4 // 5:
for k in range(32):
for l in range(32):
for m in range(3):
classified_data_2[valid_ptr][0][k][l][m] = classified_data[i][j][k][l][m]
y_data[valid_ptr] = assignments[i]
valid_ptr += 1
else:
for k in range(32):
for l in range(32):
for m in range(3):
classified_data_2[train_ptr][0][k][l][m] = classified_data[i][j][k][l][m]
y_data[train_ptr] = assignments[i]
train_ptr += 1
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
s = 0
for i in range(15):
print(i, end=" ")
for j in range(lengths[i]):
x = 0
for k in range(32):
for l in range(32):
for m in range(3):
reshaped_tr[s + j if j < lengths[i] * 4 // 5 else 36000 + s // 4 + (j - lengths[i] * 4 // 5)][x] = classified_data_2[i][j][k][l][m]
x += 1
reshaped_tr[s + j if j < lengths[i] * 4 // 5 else 36000 + s // 4 + (j - lengths[i] * 4 // 5)][3072] = i
s += lengths[i] * 4 // 5
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
df = pd.DataFrame(reshaped_tr)
df.head()
| 0 | 1 | 2 | 3 | 4 | 5 | 6 | 7 | 8 | 9 | ... | 3063 | 3064 | 3065 | 3066 | 3067 | 3068 | 3069 | 3070 | 3071 | 3072 | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| 0 | 236.0 | 238.0 | 235.0 | 213.0 | 215.0 | 212.0 | 180.0 | 182.0 | 179.0 | 159.0 | ... | 175.0 | 174.0 | 163.0 | 184.0 | 184.0 | 171.0 | 215.0 | 216.0 | 199.0 | 0.0 |
| 1 | 72.0 | 112.0 | 208.0 | 71.0 | 110.0 | 205.0 | 70.0 | 110.0 | 205.0 | 71.0 | ... | 197.0 | 198.0 | 200.0 | 206.0 | 208.0 | 205.0 | 176.0 | 180.0 | 190.0 | 0.0 |
| 2 | 185.0 | 200.0 | 241.0 | 185.0 | 200.0 | 238.0 | 188.0 | 204.0 | 238.0 | 192.0 | ... | 51.0 | 60.0 | 89.0 | 45.0 | 56.0 | 77.0 | 35.0 | 46.0 | 66.0 | 0.0 |
| 3 | 151.0 | 144.0 | 86.0 | 145.0 | 138.0 | 80.0 | 141.0 | 134.0 | 76.0 | 150.0 | ... | 135.0 | 117.0 | 71.0 | 115.0 | 99.0 | 55.0 | 118.0 | 102.0 | 63.0 | 0.0 |
| 4 | 236.0 | 221.0 | 200.0 | 230.0 | 215.0 | 194.0 | 240.0 | 224.0 | 203.0 | 248.0 | ... | 76.0 | 87.0 | 91.0 | 83.0 | 94.0 | 100.0 | 88.0 | 98.0 | 107.0 | 0.0 |
5 rows × 3073 columns
# Time for some exploratory data analysis (EDA).
df[3072].hist()
<AxesSubplot:>
Some classes contain more images than others, so that is something to watch out for.
'''# Plotting images from each class
indices = [0, 5000, 6500, 9800, 11000, 13500, 15000, 17000, 19980, 20198, 22000, 24890, 26500, 30500, 34000]
distinct = set()
for i in indices:
distinct.add(df[3072][i])
assert len(distinct) == len(indices)''';
for i in indices[1:2]:
row = df.drop(3072, axis=1)[i:i+1]
print(df[i:i+1][3072])
new_np_array = np.zeros((32, 32, 3), dtype=np.uint8)
for a in range(32):
for b in range(32):
for c in range(3):
new_np_array[a][b][c] = row[96 * a + 3 * b + c]
plt.imshow(new_np_array)
--------------------------------------------------------------------------- NameError Traceback (most recent call last) <ipython-input-27-077ace9d777d> in <module>() ----> 1 for i in indices[1:2]: 2 row = df.drop(3072, axis=1)[i:i+1] 3 print(df[i:i+1][3072]) 4 new_np_array = np.zeros((32, 32, 3), dtype=np.uint8) 5 for a in range(32): NameError: name 'indices' is not defined
I aim to construct a full convolutional neural network architecture and then tune it based on the results.
Reminder: the classes are:
1) aquatic animals (classes 1-10) 3) flowers (classes 11-15) 4) food containers (classes 16-20) 5) fruit and vegetables (classes 21-25) 6) household electrical devices (classes 26-30) 7) household furniture (classes 31-35) 8) insects (classes 36-40) 9) large carnivores (classes 41-45) 10) large man-made outdoor things (classes 46-50) NOT INCLUDED 11) large natural outdoor scenes (classes 51-55) NOT INCLUDED 12) large omnivores and herbivores (classes 56-60) 13) small to medium sized mammals (classes 61-65, 81-85) 14) non-insect invertebrates (classes 66-70) 15) people (classes 71-75) 16) reptiles (classes 76-80) 17) trees (classes 86-90) 18) vehicles (classes 91-100)
Source: https://www.researchgate.net/publication/365130408_Optimal_Design_of_Convolutional_Neural_Network_Architectures_Using_Teaching-Learning-Based_Optimization_for_Image_Classification https://medium.com/analytics-vidhya/how-relu-works-f317a947bdc6
Convolutional layer
The convolutional layer's job is to detect low-level features in the image data. In our specific case, these low-level features are important because, for instance, they can help to distinguish between household furniture (which are characterized by straight lines and right angles) and reptiles (characterized by curves).
The features can be very subtle in some cases, e.g. reptiles vs. invertebrates, so I think I will use a small kernel (3x3). This is also done to ensure that training doesn't take too long. The stride will be 1x1 to ensure that I don't miss anything.
For the activation function, I will use the ReLU function. This is because I don't think that the vanishing gradient problem is particularly problematic for this task, since the types of predictions made are quite high-level anyway. Also, as a result it wouldn't take too long for the ReLU to cause the parameters to converge (comparatively).
After the conv layer there will be pooling. Max-pooling is what I intend to use (because that way only the most important features are kept). The dimensionality of the output will be 16x16 so a 2x2 kernel will be used.
It is common in CNNs to have multiple conv layers (with multiple pooling layers as well). For this reason (and for reasons related to the data that I have observed) I will add another convolutional layer. Reason being, only the "type of object" is being predicted, so there need not be too many features. It's better to lower the number of features to prevent overfitting.
So, another convolutional layer with a 3x3 kernel and 1x1 stride, and we activate with the logistic function this time. This is to prevent there being too much dichotomy and there is the possibility for features to be "half detected". Another max-pooling afterwards.
Now come the fully connected layers. There will be three fully connected layers in the neural network.
from tensorflow.keras import layers, models
initial_cnn = models.Sequential()
# https://www.tensorflow.org/api_docs/python/tf/keras/layers
initial_cnn.add(layers.InputLayer(input_shape=(1, 3, 32, 32)))
initial_cnn.add(layers.Conv3D(activation='relu', kernel_size=3, filters=3, strides=1, padding='same'))
initial_cnn.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
initial_cnn.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=3, strides=1, padding='same'))
initial_cnn.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
initial_cnn.add(layers.Flatten())
initial_cnn.add(layers.Dense(units=64, input_dim=192, activation='relu'))
initial_cnn.add(layers.Dense(units=16, input_dim=64, activation='softmax'))
initial_cnn.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
initial_cnn.compile('sgd', 'sparse_categorical_crossentropy', ['acc'])
input_data = np.zeros((45000, 1, 3, 32, 32), dtype=np.uint8)
columns = []
for i in range(3072): columns.append(np.array(df[i]))
for i in range(45000):
x = 0
for k in range(32):
for l in range(32):
for m in range(3):
input_data[i][0][m][k][l] = columns[x][i]
x += 1
if i % 100 == 0: print(i, end=" ")
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500 15600 15700 15800 15900 16000 16100 16200 16300 16400 16500 16600 16700 16800 16900 17000 17100 17200 17300 17400 17500 17600 17700 17800 17900 18000 18100 18200 18300 18400 18500 18600 18700 18800 18900 19000 19100 19200 19300 19400 19500 19600 19700 19800 19900 20000 20100 20200 20300 20400 20500 20600 20700 20800 20900 21000 21100 21200 21300 21400 21500 21600 21700 21800 21900 22000 22100 22200 22300 22400 22500 22600 22700 22800 22900 23000 23100 23200 23300 23400 23500 23600 23700 23800 23900 24000 24100 24200 24300 24400 24500 24600 24700 24800 24900 25000 25100 25200 25300 25400 25500 25600 25700 25800 25900 26000 26100 26200 26300 26400 26500 26600 26700 26800 26900 27000 27100 27200 27300 27400 27500 27600 27700 27800 27900 28000 28100 28200 28300 28400 28500 28600 28700 28800 28900 29000 29100 29200 29300 29400 29500 29600 29700 29800 29900 30000 30100 30200 30300 30400 30500 30600 30700 30800 30900 31000 31100 31200 31300 31400 31500 31600 31700 31800 31900 32000 32100 32200 32300 32400 32500 32600 32700 32800 32900 33000 33100 33200 33300 33400 33500 33600 33700 33800 33900 34000 34100 34200 34300 34400 34500 34600 34700 34800 34900 35000 35100 35200 35300 35400 35500 35600 35700 35800 35900 36000 36100 36200 36300 36400 36500 36600 36700 36800 36900 37000 37100 37200 37300 37400 37500 37600 37700 37800 37900 38000 38100 38200 38300 38400 38500 38600 38700 38800 38900 39000 39100 39200 39300 39400 39500 39600 39700 39800 39900 40000 40100 40200 40300 40400 40500 40600 40700 40800 40900 41000 41100 41200 41300 41400 41500 41600 41700 41800 41900 42000 42100 42200 42300 42400 42500 42600 42700 42800 42900 43000 43100 43200 43300 43400 43500 43600 43700 43800 43900 44000 44100 44200 44300 44400 44500 44600 44700 44800 44900
def one_hot_encode(arr):
res=np.zeros((45000, 15), dtype=np.uint8)
for i in range(45000):
res[i][int(arr[i])]=1
return res
initial_cnn.fit(input_data, (np.array(df[3072])), batch_size=10, epochs=5, validation_split=0.2)
Train on 36000 samples, validate on 9000 samples Epoch 1/5 7970/36000 [=====>........................] - ETA: 51s - loss: 2.6828 - acc: 0.1148
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-205-47af483ba2e9> in <module>() 4 res[i][int(arr[i])]=1 5 return res ----> 6 initial_cnn.fit(input_data, (np.array(df[3072])), batch_size=10, epochs=5, validation_split=0.2) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
initial_cnn.predict(input_data[:1])
array([[0.19197462, 0.0158524 , 0.04322896, 0.01739957, 0.06896571,
0.03920421, 0.02945076, 0.03653134, 0.05463577, 0.06335519,
0.04358657, 0.03378925, 0.05564197, 0.10888346, 0.19750026]],
dtype=float32)
import tensorflow as tf
import tensorflow.keras as keras
max(initial_cnn.history.history['val_acc'])
0.1843333350204759
Slight issue: the best performance is 18.4%
We need to improve the neural network, and one thing that needs to be improved is to add a learning rate (which needs to be tuned) and change the architecture as well.
# Run a grid search with powers of 2 (goes into negative exponents)
search_space = []
for i in range(-5, 6):
search_space.append(2**i)
accs = []
for val in search_space:
cur_cnn = models.Sequential()
cur_cnn.add(layers.InputLayer(input_shape=(32, 32, 3)))
cur_cnn.add(layers.Conv2D(activation='relu', kernel_size=3, filters=3, strides=1, padding='same'))
cur_cnn.add(layers.MaxPool2D(pool_size=2, strides=2, padding='same'))
cur_cnn.add(layers.Conv2D(activation='softmax', kernel_size=3, filters=3, strides=1, padding='same'))
cur_cnn.add(layers.MaxPool2D(pool_size=2, strides=2, padding='same'))
cur_cnn.add(layers.Flatten())
cur_cnn.add(layers.Dense(units=64, input_dim=192, activation='relu'))
cur_cnn.add(layers.Dense(units=16, input_dim=64, activation='softmax'))
cur_cnn.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.SGD(lr=val)
cur_cnn.compile(opt, 'sparse_categorical_crossentropy', ['acc'])
cur_cnn.fit(input_data, (np.array(df[3072])), batch_size=100, epochs=5, validation_split=0.2)
accs.append(max(cur_cnn.history.history['val_acc']))
print(accs)
Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 279us/step - loss: 2.6805 - acc: 0.1077 - val_loss: 2.6638 - val_acc: 0.1137 Epoch 2/5 36000/36000 [==============================] - 10s 285us/step - loss: 2.6611 - acc: 0.1160 - val_loss: 2.6598 - val_acc: 0.1190 Epoch 3/5 36000/36000 [==============================] - 12s 322us/step - loss: 2.6594 - acc: 0.1158 - val_loss: 2.6591 - val_acc: 0.1130 Epoch 4/5 36000/36000 [==============================] - 10s 288us/step - loss: 2.6586 - acc: 0.1191 - val_loss: 2.6586 - val_acc: 0.1250 Epoch 5/5 36000/36000 [==============================] - 10s 265us/step - loss: 2.6581 - acc: 0.1259 - val_loss: 2.6581 - val_acc: 0.1274 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 276us/step - loss: 2.6726 - acc: 0.1067 - val_loss: 2.6601 - val_acc: 0.1176 Epoch 2/5 36000/36000 [==============================] - 10s 264us/step - loss: 2.6598 - acc: 0.1140 - val_loss: 2.6592 - val_acc: 0.1142 Epoch 3/5 36000/36000 [==============================] - 9s 258us/step - loss: 2.6594 - acc: 0.1158 - val_loss: 2.6589 - val_acc: 0.1251 Epoch 4/5 36000/36000 [==============================] - 10s 270us/step - loss: 2.6587 - acc: 0.1125 - val_loss: 2.6574 - val_acc: 0.1121 Epoch 5/5 36000/36000 [==============================] - 10s 266us/step - loss: 2.6532 - acc: 0.1228 - val_loss: 2.6419 - val_acc: 0.1282 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 277us/step - loss: 2.6651 - acc: 0.1102 - val_loss: 2.6586 - val_acc: 0.1079 Epoch 2/5 36000/36000 [==============================] - 10s 276us/step - loss: 2.6557 - acc: 0.1185 - val_loss: 2.6498 - val_acc: 0.1246 Epoch 3/5 36000/36000 [==============================] - 10s 276us/step - loss: 2.6289 - acc: 0.1409 - val_loss: 2.6473 - val_acc: 0.1214 Epoch 4/5 36000/36000 [==============================] - 10s 275us/step - loss: 2.5933 - acc: 0.1581 - val_loss: 2.5949 - val_acc: 0.1534 Epoch 5/5 36000/36000 [==============================] - 10s 279us/step - loss: 2.5558 - acc: 0.1745 - val_loss: 2.5471 - val_acc: 0.1729 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 275us/step - loss: 2.6470 - acc: 0.1189 - val_loss: 2.6003 - val_acc: 0.1428 Epoch 2/5 36000/36000 [==============================] - 10s 267us/step - loss: 2.5934 - acc: 0.1418 - val_loss: 2.5756 - val_acc: 0.1421 Epoch 3/5 36000/36000 [==============================] - 10s 289us/step - loss: 2.5614 - acc: 0.1579 - val_loss: 2.5357 - val_acc: 0.1620 Epoch 4/5 36000/36000 [==============================] - 9s 262us/step - loss: 2.5350 - acc: 0.1714 - val_loss: 2.5349 - val_acc: 0.1689 Epoch 5/5 36000/36000 [==============================] - 10s 272us/step - loss: 2.5165 - acc: 0.1797 - val_loss: 2.5244 - val_acc: 0.1730 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 11s 302us/step - loss: 2.6364 - acc: 0.1316 - val_loss: 2.6073 - val_acc: 0.1234 Epoch 2/5 36000/36000 [==============================] - 9s 261us/step - loss: 2.5523 - acc: 0.1746 - val_loss: 2.5104 - val_acc: 0.1912 Epoch 3/5 36000/36000 [==============================] - 10s 269us/step - loss: 2.5084 - acc: 0.1890 - val_loss: 2.5156 - val_acc: 0.1893 Epoch 4/5 36000/36000 [==============================] - 9s 256us/step - loss: 2.4894 - acc: 0.1961 - val_loss: 2.4486 - val_acc: 0.2057 Epoch 5/5 36000/36000 [==============================] - 9s 262us/step - loss: 2.5233 - acc: 0.1797 - val_loss: 2.5116 - val_acc: 0.1673 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 280us/step - loss: 2.6630 - acc: 0.1109 - val_loss: 2.6628 - val_acc: 0.1111 Epoch 2/5 36000/36000 [==============================] - 9s 255us/step - loss: 2.6618 - acc: 0.1121 - val_loss: 2.6613 - val_acc: 0.1112 Epoch 3/5 36000/36000 [==============================] - 10s 267us/step - loss: 2.6501 - acc: 0.1164 - val_loss: 2.6940 - val_acc: 0.0984 Epoch 4/5 36000/36000 [==============================] - 9s 256us/step - loss: 2.6136 - acc: 0.1354 - val_loss: 2.5648 - val_acc: 0.1731 Epoch 5/5 36000/36000 [==============================] - 10s 265us/step - loss: 2.5585 - acc: 0.1641 - val_loss: 2.5670 - val_acc: 0.1443 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 285us/step - loss: 2.6656 - acc: 0.1124 - val_loss: 2.6649 - val_acc: 0.1117 Epoch 2/5 36000/36000 [==============================] - 9s 262us/step - loss: 2.6647 - acc: 0.1112 - val_loss: 2.6638 - val_acc: 0.1106 Epoch 3/5 36000/36000 [==============================] - 10s 269us/step - loss: 2.6646 - acc: 0.1124 - val_loss: 2.6656 - val_acc: 0.1111 Epoch 4/5 36000/36000 [==============================] - 9s 260us/step - loss: 2.6651 - acc: 0.1096 - val_loss: 2.6654 - val_acc: 0.1111 Epoch 5/5 36000/36000 [==============================] - 10s 274us/step - loss: 2.6652 - acc: 0.1107 - val_loss: 2.6649 - val_acc: 0.1111 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 288us/step - loss: 2.6715 - acc: 0.1108 - val_loss: 2.6848 - val_acc: 0.1111 Epoch 2/5 36000/36000 [==============================] - 9s 262us/step - loss: 2.6700 - acc: 0.1131 - val_loss: 2.6691 - val_acc: 0.1111 Epoch 3/5 36000/36000 [==============================] - 10s 277us/step - loss: 2.6715 - acc: 0.1126 - val_loss: 2.6693 - val_acc: 0.1111 Epoch 4/5 36000/36000 [==============================] - 9s 254us/step - loss: 2.6708 - acc: 0.1095 - val_loss: 2.6777 - val_acc: 0.1111 Epoch 5/5 36000/36000 [==============================] - 9s 260us/step - loss: 2.6708 - acc: 0.1106 - val_loss: 2.6702 - val_acc: 0.1111 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 11s 295us/step - loss: 2.6904 - acc: 0.1087 - val_loss: 2.6726 - val_acc: 0.1111 Epoch 2/5 36000/36000 [==============================] - 9s 258us/step - loss: 2.6894 - acc: 0.1101 - val_loss: 2.6826 - val_acc: 0.1111 Epoch 3/5 36000/36000 [==============================] - 10s 269us/step - loss: 2.6893 - acc: 0.1078 - val_loss: 2.6891 - val_acc: 0.1111 Epoch 4/5 36000/36000 [==============================] - 9s 261us/step - loss: 2.6909 - acc: 0.1099 - val_loss: 2.7099 - val_acc: 0.1111 Epoch 5/5 36000/36000 [==============================] - 10s 288us/step - loss: 2.6897 - acc: 0.1093 - val_loss: 2.6820 - val_acc: 0.1111 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 10s 288us/step - loss: 7.7178 - acc: 0.0609 - val_loss: 8.1408 - val_acc: 0.0556 Epoch 2/5 36000/36000 [==============================] - 9s 258us/step - loss: 8.1240 - acc: 0.0550 - val_loss: 8.1290 - val_acc: 0.0556 Epoch 3/5 36000/36000 [==============================] - 10s 276us/step - loss: 8.1391 - acc: 0.0542 - val_loss: 8.2519 - val_acc: 0.0556 Epoch 4/5 36000/36000 [==============================] - 10s 272us/step - loss: 8.1275 - acc: 0.0539 - val_loss: 8.1813 - val_acc: 0.0556 Epoch 5/5 36000/36000 [==============================] - 9s 263us/step - loss: 8.1617 - acc: 0.0552 - val_loss: 8.4297 - val_acc: 0.0556 Train on 36000 samples, validate on 9000 samples Epoch 1/5 36000/36000 [==============================] - 11s 292us/step - loss: 13.5795 - acc: 0.0568 - val_loss: 13.6592 - val_acc: 0.0556 Epoch 2/5 36000/36000 [==============================] - 10s 277us/step - loss: 13.9051 - acc: 0.0576 - val_loss: 14.1076 - val_acc: 0.0556 Epoch 3/5 36000/36000 [==============================] - 10s 286us/step - loss: 13.9271 - acc: 0.0545 - val_loss: 14.1658 - val_acc: 0.0556 Epoch 4/5 36000/36000 [==============================] - 9s 262us/step - loss: 13.9205 - acc: 0.0553 - val_loss: 14.2486 - val_acc: 0.0556 Epoch 5/5 36000/36000 [==============================] - 9s 258us/step - loss: 13.9093 - acc: 0.0558 - val_loss: 14.2010 - val_acc: 0.0556 [0.1274444436447488, 0.12822222198463148, 0.1728888902399275, 0.17299999921686118, 0.2056666663951344, 0.1731111099322637, 0.11166666665424903, 0.1111111111111111, 0.1111111111111111, 0.05555555555555555, 0.05555555555555555]
plt.plot(accs)
[<matplotlib.lines.Line2D at 0x26dd8288588>]
It can be seen that when mu (learning rate) is equal to 0.5, the model achieves the best accuracy. Notice how at higher learning rates, the model degrades to a dummy classifier (predicting the most frequent one), and at even higher learning rates it would appear that it bound itself to some infrequent category (for whatever reason). An accuracy of 0.21 is the best we have yet, which is still pretty bad but it can be improved through other means.
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 3, 32, 32)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=3, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=3, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=16, input_dim=768, activation='softmax'))
second_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.SGD(lr=0.3) # following results from grid search, not necessarily best for this model
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(input_data, one_hot_encode(np.array(df[3072])), batch_size=100, epochs=20, validation_split=0.2)
Train on 36000 samples, validate on 9000 samples Epoch 1/20 36000/36000 [==============================] - 19s 524us/step - loss: 2.6513 - acc: 0.1167 - val_loss: 2.6282 - val_acc: 0.1520 Epoch 2/20 36000/36000 [==============================] - 16s 434us/step - loss: 2.6254 - acc: 0.1377 - val_loss: 2.6264 - val_acc: 0.1386 Epoch 3/20 36000/36000 [==============================] - 17s 464us/step - loss: 2.6440 - acc: 0.1285 - val_loss: 2.6133 - val_acc: 0.1297 Epoch 4/20 36000/36000 [==============================] - 14s 402us/step - loss: 2.6013 - acc: 0.1480 - val_loss: 2.5909 - val_acc: 0.1440 Epoch 5/20 36000/36000 [==============================] - 15s 403us/step - loss: 2.6162 - acc: 0.1398 - val_loss: 2.6438 - val_acc: 0.1111 Epoch 6/20 20900/36000 [================>.............] - ETA: 5s - loss: 2.6337 - acc: 0.1356
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-207-81bf64c7de2c> in <module>() 13 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 14 ---> 15 second_iteration.fit(input_data, one_hot_encode(np.array(df[3072])), batch_size=100, epochs=20, validation_split=0.2) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
New best validation accuracy: 22.7%. It's a step in the right direction to replace "sparse_categorical_crossentropy" with "categorical_crossentropy" to ensure that the relative probabilities get taken into account, because objects can fall into "gray areas" and there are multiple different types that can look similar to each other. A goal is to get the model to predict the right "even higher-level" type of an object, and so this is better.
On a related note, I wonder whether there is a loss function that takes into account differing levels of similarity between "object types".
'''third_iteration = models.Sequential()
third_iteration.add(layers.InputLayer(input_shape=(32, 32, 3)))
third_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=3, strides=1, padding='same'))
third_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=3, strides=1, padding='same'))
third_iteration.add(layers.MaxPool2D(pool_size=2, strides=2, padding='same'))
third_iteration.add(layers.Flatten())
third_iteration.add(layers.Dense(units=16, input_dim=768, activation='softmax'))
third_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.SGD(lr=0.1) # following results from grid search, not necessarily best for this model
third_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
third_iteration.fit(input_data, one_hot_encode(np.array(df[3072])), batch_size=100, epochs=20, validation_split=0.2)'''
None
Quite frankly, no matter how I change the paramters, the accuracy is depressingly bad.
I have come to the conclusion that I have to examine the underlying architecture of the model and make changes according to the dataset. EDA is hard to do with this type of data, but I nevertheless found that:
We therefore need a convolutional step capable of detecting the rather sharp identifying features but not the noise. We need a sort of dichotomy in our feature extraction.
This could be achieved using, for instance, the sigmoid function. But this function is not suited for classification, so let's use softmax for activation in Conv2D.
'''third_iteration = models.Sequential()
third_iteration.add(layers.InputLayer(input_shape=(32, 32, 3)))
third_iteration.add(layers.Conv2D(activation='softmax', kernel_size=5, filters=3, strides=1, padding='same'))
third_iteration.add(layers.MaxPool2D(pool_size=2, strides=2, padding='same'))
third_iteration.add(layers.Flatten())
third_iteration.add(layers.Dense(units=16, input_dim=768, activation='softmax'))
third_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.SGD(lr=0.2) # following results from grid search, not necessarily best for this model
third_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
third_iteration.fit(input_data, one_hot_encode(np.array(df[3072])), batch_size=100, epochs=20, validation_split=0.2)''';
# EDA for "white spots"
cnt = []
for i in range(15):
val = 0
for j in range(lengths[i]):
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][j][k][l][0],classified_data_2[i][j][k][l][1],classified_data_2[i][j][k][l][2],\
classified_data_2[i][j][k+1][l][0],classified_data_2[i][j][k+1][l][1],classified_data_2[i][j][k+1][l][2],\
classified_data_2[i][j][k][l+1][0],classified_data_2[i][j][k][l+1][1],classified_data_2[i][j][k][l+1][2],\
classified_data_2[i][j][k+1][l+1][0],classified_data_2[i][j][k+1][l+1][1],classified_data_2[i][j][k+1][l+1][2])==255:
found = True
break
if found: break
if found: val += 1
if j % 100 == 0: print(j, end=" ")
cnt.append(val)
print(i, end=" ")
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 0 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 1 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 3 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 4 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 5 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 6 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 7 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 8 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 9 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 10 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 11 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 12 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 13 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 14
cnt
[262, 73, 336, 394, 657, 456, 235, 55, 87, 133, 179, 191, 151, 164, 633]
It can be seen that each category has 50 or more images that contain a 2x2 white area. While this is not a perfect measure of whether or not the image has a white background, it is good enough for our purposes. If we took 50 per class we would have a dataset of 750 images which is far less than the 45000 we started with, but it's not nothing, and we could try fitting a CNN to it to see what would happen.
The reasoning is that CNN is picking up on far too much background noise (that's my hypothesis) so it would be nice to test it out.
good_data = np.zeros((825, 1, 3, 32, 32), dtype=np.uint8)
ptr = 0
for i in range(15):
val = 0
for j in range(lengths[i]):
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][j][k][l][0],classified_data_2[i][j][k][l][1],classified_data_2[i][j][k][l][2],\
classified_data_2[i][j][k+1][l][0],classified_data_2[i][j][k+1][l][1],classified_data_2[i][j][k+1][l][2],\
classified_data_2[i][j][k][l+1][0],classified_data_2[i][j][k][l+1][1],classified_data_2[i][j][k][l+1][2],\
classified_data_2[i][j][k+1][l+1][0],classified_data_2[i][j][k+1][l+1][1],classified_data_2[i][j][k+1][l+1][2])==255:
found = True
break
if found: break
if found:
if val == 55: break
val += 1
for k in range(3):
for l in range(32):
for m in range(32):
good_data[ptr][0][k][l][m] = classified_data_2[i][j][l][m][k]
ptr += 1
if j % 100 == 0: print(j, end=" ")
print(i, end=" ")
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 0 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1 0 100 200 300 400 2 0 100 200 300 3 0 100 200 4 0 100 200 5 0 100 200 300 400 500 6 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 7 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 8 0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 9 0 100 200 300 400 500 600 700 800 10 0 100 200 300 400 500 600 700 11 0 100 200 300 400 500 600 700 800 900 1000 12 0 100 200 300 400 500 600 700 800 900 13 0 100 200 300 400 500 14
white_tr = np.zeros((660, 1, 3, 32, 32), dtype=np.uint8)
white_valid = np.zeros((165, 1, 3, 32, 32), dtype=np.uint8)
ptr_tr = 0
ptr_valid = 0
for i in range(15):
for j in range(55):
if j >= 44:
for k in range(3):
for l in range(32):
for m in range(32):
white_valid[ptr_valid][0][k][l][m] = good_data[i * 55 + j][0][k][l][m]
ptr_valid += 1
else:
for k in range(3):
for l in range(32):
for m in range(32):
white_tr[ptr_tr][0][k][l][m] = good_data[i * 55 + j][0][k][l][m]
ptr_tr += 1
print(i, end=" ")
from random import shuffle
tr_shuffle = [*range(660)]
valid_shuffle = [*range(165)]
shuffle(tr_shuffle)
shuffle(valid_shuffle)
white_data = np.zeros((825, 1, 3, 32, 32), dtype=np.uint8)
for i in range(660):
for k in range(3):
for l in range(32):
for m in range(32):
white_data[tr_shuffle[i]][0][k][l][m] = white_tr[i][0][k][l][m]
for i in range(165):
for k in range(3):
for l in range(32):
for m in range(32):
white_data[valid_shuffle[i] + 660][0][k][l][m] = white_valid[i][0][k][l][m]
white_y = np.zeros(825, dtype=np.uint8)
for i in range(15):
for j in range(44 * i, 44 * i + 44):
white_y[tr_shuffle[j]] = i
for j in range(660 + 11 * i, 660 + 11 * i + 11):
white_y[valid_shuffle[j - 660] + 660] = i
0 1 2 3 4 5 6 7 8 9 10 11 12 13 14
from random import randint
ind = randint(0, 660)
plt.imshow(np.transpose(white_data[ind][0], (1, 2, 0)))
white_y[ind]
13
transposed_white_data = np.zeros((825, 1, 32, 32, 3), dtype=np.uint8)
for i in range(825):
for k in range(3):
for l in range(32):
for m in range(32):
transposed_white_data[i][0][l][m][k] = 255 - white_data[i][0][k][l][m]
if i % 100 == 0: print(i, end=" ")
0 100 200 300 400 500 600 700 800
def one_hot_encode(arr):
res=np.zeros((arr.size, 15), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
# Run a grid search with powers of 2 (goes into negative exponents)
search_space = []
for i in range(-5, 6):
search_space.append(2**i)
accs = []
for val in search_space:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=16, input_dim=768, activation='softmax'))
second_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.SGD(lr=val)
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=10, validation_split=0.2)
accs.append(max(second_iteration.history.history['val_acc']))
print(accs)
Train on 660 samples, validate on 165 samples Epoch 1/10 660/660 [==============================] - 11s 17ms/step - loss: 2.7117 - acc: 0.0667 - val_loss: 2.7079 - val_acc: 0.0727 Epoch 2/10 660/660 [==============================] - 4s 6ms/step - loss: 2.7098 - acc: 0.0682 - val_loss: 2.7072 - val_acc: 0.0788 Epoch 3/10 660/660 [==============================] - 4s 6ms/step - loss: 2.7074 - acc: 0.0667 - val_loss: 2.7042 - val_acc: 0.0788 Epoch 4/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7054 - acc: 0.0879 - val_loss: 2.7016 - val_acc: 0.0727 Epoch 5/10 660/660 [==============================] - 4s 6ms/step - loss: 2.7023 - acc: 0.0818 - val_loss: 2.6989 - val_acc: 0.0848 Epoch 6/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6990 - acc: 0.0818 - val_loss: 2.6956 - val_acc: 0.0970 Epoch 7/10 660/660 [==============================] - 4s 7ms/step - loss: 2.6957 - acc: 0.0727 - val_loss: 2.6919 - val_acc: 0.0788 Epoch 8/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6926 - acc: 0.0742 - val_loss: 2.6882 - val_acc: 0.0788 Epoch 9/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6877 - acc: 0.0848 - val_loss: 2.6828 - val_acc: 0.0848 Epoch 10/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6821 - acc: 0.0788 - val_loss: 2.6789 - val_acc: 0.0909 Train on 660 samples, validate on 165 samples Epoch 1/10 660/660 [==============================] - 11s 17ms/step - loss: 2.7119 - acc: 0.0591 - val_loss: 2.6999 - val_acc: 0.0667 Epoch 2/10 660/660 [==============================] - 4s 6ms/step - loss: 2.7020 - acc: 0.0576 - val_loss: 2.6904 - val_acc: 0.0848 Epoch 3/10 660/660 [==============================] - 4s 7ms/step - loss: 2.6899 - acc: 0.0924 - val_loss: 2.6822 - val_acc: 0.1394 Epoch 4/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6781 - acc: 0.1000 - val_loss: 2.6690 - val_acc: 0.1333 Epoch 5/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6643 - acc: 0.1227 - val_loss: 2.7183 - val_acc: 0.0788 Epoch 6/10 660/660 [==============================] - 4s 6ms/step - loss: 2.7084 - acc: 0.0788 - val_loss: 2.7034 - val_acc: 0.1091 Epoch 7/10 660/660 [==============================] - 4s 7ms/step - loss: 2.6759 - acc: 0.1152 - val_loss: 2.6554 - val_acc: 0.1515 Epoch 8/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6409 - acc: 0.1348 - val_loss: 2.6415 - val_acc: 0.1576 Epoch 9/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6298 - acc: 0.1439 - val_loss: 2.6244 - val_acc: 0.1515 Epoch 10/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6080 - acc: 0.1515 - val_loss: 2.6137 - val_acc: 0.1515 Train on 660 samples, validate on 165 samples Epoch 1/10 660/660 [==============================] - 11s 17ms/step - loss: 2.7165 - acc: 0.0652 - val_loss: 2.7069 - val_acc: 0.0848 Epoch 2/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7149 - acc: 0.0576 - val_loss: 2.7052 - val_acc: 0.0848 Epoch 3/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7118 - acc: 0.0773 - val_loss: 2.7003 - val_acc: 0.1152 Epoch 4/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7020 - acc: 0.0712 - val_loss: 2.6998 - val_acc: 0.0909 Epoch 5/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6869 - acc: 0.1182 - val_loss: 2.6765 - val_acc: 0.1394 Epoch 6/10 660/660 [==============================] - 4s 7ms/step - loss: 2.6616 - acc: 0.1424 - val_loss: 2.6478 - val_acc: 0.1515 Epoch 7/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6448 - acc: 0.1318 - val_loss: 2.6402 - val_acc: 0.1515 Epoch 8/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6102 - acc: 0.1667 - val_loss: 2.6181 - val_acc: 0.1333 Epoch 9/10 660/660 [==============================] - 4s 6ms/step - loss: 2.5847 - acc: 0.1606 - val_loss: 2.5935 - val_acc: 0.2000 Epoch 10/10 660/660 [==============================] - 4s 7ms/step - loss: 2.5735 - acc: 0.1712 - val_loss: 2.5829 - val_acc: 0.1697 Train on 660 samples, validate on 165 samples Epoch 1/10 660/660 [==============================] - 11s 17ms/step - loss: 2.7225 - acc: 0.0545 - val_loss: 2.7089 - val_acc: 0.0727 Epoch 2/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7186 - acc: 0.0530 - val_loss: 2.7030 - val_acc: 0.0727 Epoch 3/10 660/660 [==============================] - 4s 6ms/step - loss: 2.7023 - acc: 0.0833 - val_loss: 2.6715 - val_acc: 0.1515 Epoch 4/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6822 - acc: 0.1091 - val_loss: 2.6556 - val_acc: 0.1152 Epoch 5/10 660/660 [==============================] - 4s 7ms/step - loss: 2.6430 - acc: 0.1227 - val_loss: 2.6479 - val_acc: 0.1212 Epoch 6/10 660/660 [==============================] - 4s 7ms/step - loss: 2.6615 - acc: 0.1091 - val_loss: 2.6660 - val_acc: 0.0909 Epoch 7/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6459 - acc: 0.1212 - val_loss: 2.7331 - val_acc: 0.0848 Epoch 8/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6318 - acc: 0.1182 - val_loss: 2.7977 - val_acc: 0.0667 Epoch 9/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7618 - acc: 0.0591 - val_loss: 2.7181 - val_acc: 0.0667 Epoch 10/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7281 - acc: 0.0439 - val_loss: 2.7101 - val_acc: 0.0667 Train on 660 samples, validate on 165 samples Epoch 1/10 660/660 [==============================] - 11s 17ms/step - loss: 2.7409 - acc: 0.0667 - val_loss: 2.7427 - val_acc: 0.0667 Epoch 2/10 660/660 [==============================] - 4s 7ms/step - loss: 2.7419 - acc: 0.0515 - val_loss: 2.7151 - val_acc: 0.0606 Epoch 3/10 480/660 [====================>.........] - ETA: 1s - loss: 2.7414 - acc: 0.0521
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-342-069475d966b7> in <module>() 26 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 27 ---> 28 second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=10, validation_split=0.2) 29 accs.append(max(second_iteration.history.history['val_acc'])) 30 print(accs) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=16, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=16, input_dim=768, activation='softmax'))
second_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.SGD(lr=0.125)
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=10, validation_split=0.2, shuffle=False)
Train on 660 samples, validate on 165 samples Epoch 1/10 660/660 [==============================] - 12s 19ms/step - loss: 2.7150 - acc: 0.0606 - val_loss: 2.7061 - val_acc: 0.0909 Epoch 2/10 660/660 [==============================] - 4s 6ms/step - loss: 2.7121 - acc: 0.0667 - val_loss: 2.7014 - val_acc: 0.0788 Epoch 3/10 660/660 [==============================] - 4s 5ms/step - loss: 2.7031 - acc: 0.0621 - val_loss: 2.6918 - val_acc: 0.1333 Epoch 4/10 660/660 [==============================] - 4s 5ms/step - loss: 2.6893 - acc: 0.0848 - val_loss: 2.6931 - val_acc: 0.1212 Epoch 5/10 660/660 [==============================] - 4s 5ms/step - loss: 2.6768 - acc: 0.0864 - val_loss: 2.6762 - val_acc: 0.1091 Epoch 6/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6460 - acc: 0.1591 - val_loss: 2.6624 - val_acc: 0.1636 Epoch 7/10 660/660 [==============================] - 4s 6ms/step - loss: 2.6116 - acc: 0.1773 - val_loss: 2.6303 - val_acc: 0.1455 Epoch 8/10 660/660 [==============================] - 4s 5ms/step - loss: 2.5939 - acc: 0.1894 - val_loss: 2.6298 - val_acc: 0.1333 Epoch 9/10 660/660 [==============================] - 3s 5ms/step - loss: 2.5987 - acc: 0.1545 - val_loss: 2.6380 - val_acc: 0.1091 Epoch 10/10 660/660 [==============================] - 4s 6ms/step - loss: 2.5991 - acc: 0.1303 - val_loss: 2.6272 - val_acc: 0.1030
I will recollect my thoughts here.
I was able to make some progress with training a CNN on white-background-only images. My reasoning was that the background noise would not be detected, so a simple ReLU could eliminate all the background, and then the model could focus more on extracting actual features.
But its best training accuracy is only 22.8%, which is still very low. It might be useful to look at what the CNN is thinking (visualize a feature map).
from tensorflow.keras import Model
testing_features = Model(inputs=second_iteration.inputs, outputs=second_iteration.layers[1].output)
feature_maps = testing_features.predict(transposed_white_data[299:300])
ix = 1
for i in range(4):
for j in range(4):
# specify subplot and turn of axis
ax = plt.subplot(4, 4, ix)
ax.set_xticks([])
ax.set_yticks([])
# plot filter channel in grayscale
plt.imshow(np.transpose(feature_maps[0][0], (2, 0, 1))[ix-1], cmap='gray')
ix += 1
# show the figure
plt.show()
plt.imshow(np.transpose(white_data[306][0], (1, 2, 0)))
<matplotlib.image.AxesImage at 0x26e3c5d9940>
second_iteration.predict(transposed_white_data[306:307])
array([[0.06627624, 0.06007253, 0.05357749, 0.07312821, 0.06471668,
0.06457517, 0.06810883, 0.08241225, 0.0668057 , 0.06880648,
0.05880767, 0.06240244, 0.06656296, 0.06455852, 0.07918887]],
dtype=float32)
I've noticed that a lot of the time the model fails to get past some distinct boundary (something like 20%) so I wonder if it is converging towards a local, but not global, minimum. The categorical cross entropy loss function is based on logarithms of the probabilities, so it penalizes the model for getting the right answer but not being confident about it. Nevertheless, the penalty is even more harsh when we apply a softmax transformation to the output vector before categorical cross-entropy. In my view, this should force the model to learn more because often, it is completely indecisive, as seen just above.
Overall, the loss function that I used seems rather unsatisfactory and doesn't seem to have forced the model into learning as much. Let's define our own loss. It is based on the "softmax loss" that comes from one of the sources I cited in my process journal.
import tensorflow.keras.losses as losses
import tensorflow.keras.activations as activations
def softmax_crossentropy(y_true, y_pred):
return losses.categorical_crossentropy(y_true, activations.softmax(y_pred))
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=16, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=16, input_dim=768, activation='softmax'))
second_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=20, validation_split=0.2, shuffle=False)
Train on 660 samples, validate on 165 samples Epoch 1/20 660/660 [==============================] - 12s 19ms/step - loss: 2.7116 - acc: 0.0682 - val_loss: 2.6937 - val_acc: 0.1152 Epoch 2/20 660/660 [==============================] - 3s 5ms/step - loss: 2.6769 - acc: 0.1106 - val_loss: 2.7051 - val_acc: 0.0909 Epoch 3/20 660/660 [==============================] - 4s 6ms/step - loss: 2.6732 - acc: 0.1242 - val_loss: 2.6595 - val_acc: 0.1212 Epoch 4/20 660/660 [==============================] - 4s 5ms/step - loss: 2.6170 - acc: 0.1955 - val_loss: 2.6400 - val_acc: 0.1636 Epoch 5/20 660/660 [==============================] - 4s 5ms/step - loss: 2.5862 - acc: 0.2227 - val_loss: 2.6366 - val_acc: 0.1273 Epoch 6/20 660/660 [==============================] - 3s 5ms/step - loss: 2.5562 - acc: 0.2288 - val_loss: 2.6113 - val_acc: 0.1697 Epoch 7/20 660/660 [==============================] - 4s 5ms/step - loss: 2.5083 - acc: 0.3015 - val_loss: 2.5843 - val_acc: 0.2485 Epoch 8/20 660/660 [==============================] - 4s 5ms/step - loss: 2.4565 - acc: 0.3621 - val_loss: 2.5604 - val_acc: 0.2424 Epoch 9/20 660/660 [==============================] - 4s 5ms/step - loss: 2.4218 - acc: 0.3727 - val_loss: 2.5618 - val_acc: 0.2303 Epoch 10/20 660/660 [==============================] - 4s 5ms/step - loss: 2.3789 - acc: 0.3909 - val_loss: 2.5351 - val_acc: 0.2000 Epoch 11/20 660/660 [==============================] - 3s 5ms/step - loss: 2.3432 - acc: 0.4121 - val_loss: 2.5192 - val_acc: 0.2303 Epoch 12/20 660/660 [==============================] - 4s 6ms/step - loss: 2.3001 - acc: 0.4303 - val_loss: 2.5085 - val_acc: 0.2242 Epoch 13/20 660/660 [==============================] - 4s 5ms/step - loss: 2.2481 - acc: 0.4439 - val_loss: 2.4989 - val_acc: 0.2242 Epoch 14/20 660/660 [==============================] - 3s 5ms/step - loss: 2.2078 - acc: 0.4621 - val_loss: 2.5054 - val_acc: 0.2061 Epoch 15/20 660/660 [==============================] - 4s 5ms/step - loss: 2.1721 - acc: 0.4864 - val_loss: 2.5075 - val_acc: 0.2303 Epoch 16/20 660/660 [==============================] - 4s 6ms/step - loss: 2.1150 - acc: 0.5000 - val_loss: 2.4629 - val_acc: 0.2364 Epoch 17/20 660/660 [==============================] - 4s 5ms/step - loss: 2.0691 - acc: 0.5167 - val_loss: 2.4712 - val_acc: 0.2545 Epoch 18/20 660/660 [==============================] - 4s 5ms/step - loss: 2.0251 - acc: 0.5258 - val_loss: 2.4654 - val_acc: 0.2364 Epoch 19/20 660/660 [==============================] - 4s 5ms/step - loss: 1.9816 - acc: 0.5303 - val_loss: 2.4516 - val_acc: 0.2545 Epoch 20/20 660/660 [==============================] - 4s 5ms/step - loss: 1.9402 - acc: 0.5333 - val_loss: 2.4589 - val_acc: 0.2364
As it would turn out, the solution to our problem is surprisingly simple: use the Adam optimizer!
The difference between SGD and Adam is that Adam dynamically computes learning rates as the training goes on. I decided to give it a go just after reading about it, because of my suspicion about what was happening with SGD, on account of the fact that setting an individual learning rate for various different levels of feature specificity (at a high-level, rigid lines probably means furniture/electronics, whereas lots of spikes could mean quite a few things) probably wouldn't work. And suddenly, after 20 epochs a 53% accuracy on the training set was achieved!
But for some reason its validation accuracy is 25%. Overfitting ever?
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=16, input_dim=768, activation='softmax'))
second_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=50, validation_split=0.2, shuffle=False)
Train on 660 samples, validate on 165 samples Epoch 1/50 660/660 [==============================] - 17s 26ms/step - loss: 2.7130 - acc: 0.0848 - val_loss: 2.6932 - val_acc: 0.1152 Epoch 2/50 660/660 [==============================] - 5s 7ms/step - loss: 2.6837 - acc: 0.1242 - val_loss: 2.6700 - val_acc: 0.1333 Epoch 3/50 660/660 [==============================] - 3s 4ms/step - loss: 2.6497 - acc: 0.1197 - val_loss: 2.6439 - val_acc: 0.1091 Epoch 4/50 660/660 [==============================] - 3s 4ms/step - loss: 2.6101 - acc: 0.1636 - val_loss: 2.6248 - val_acc: 0.1818 Epoch 5/50 660/660 [==============================] - 2s 4ms/step - loss: 2.5716 - acc: 0.2227 - val_loss: 2.6084 - val_acc: 0.2061 Epoch 6/50 660/660 [==============================] - 3s 4ms/step - loss: 2.5438 - acc: 0.2652 - val_loss: 2.5859 - val_acc: 0.2303 Epoch 7/50 660/660 [==============================] - 3s 4ms/step - loss: 2.5063 - acc: 0.2909 - val_loss: 2.5766 - val_acc: 0.2303 Epoch 8/50 660/660 [==============================] - 2s 4ms/step - loss: 2.4755 - acc: 0.2985 - val_loss: 2.5712 - val_acc: 0.2000 Epoch 9/50 660/660 [==============================] - 2s 4ms/step - loss: 2.4498 - acc: 0.3136 - val_loss: 2.5435 - val_acc: 0.2000 Epoch 10/50 660/660 [==============================] - 2s 4ms/step - loss: 2.4115 - acc: 0.3273 - val_loss: 2.5310 - val_acc: 0.2303 Epoch 11/50 660/660 [==============================] - 2s 4ms/step - loss: 2.3778 - acc: 0.3455 - val_loss: 2.5270 - val_acc: 0.2182 Epoch 12/50 660/660 [==============================] - 2s 4ms/step - loss: 2.3465 - acc: 0.3545 - val_loss: 2.5176 - val_acc: 0.2182 Epoch 13/50 660/660 [==============================] - 3s 4ms/step - loss: 2.3187 - acc: 0.3545 - val_loss: 2.5068 - val_acc: 0.2121 Epoch 14/50 660/660 [==============================] - 3s 4ms/step - loss: 2.2945 - acc: 0.3621 - val_loss: 2.5191 - val_acc: 0.1939 Epoch 15/50 660/660 [==============================] - 2s 4ms/step - loss: 2.2674 - acc: 0.3742 - val_loss: 2.5073 - val_acc: 0.2061 Epoch 16/50 660/660 [==============================] - 2s 4ms/step - loss: 2.2313 - acc: 0.3939 - val_loss: 2.4887 - val_acc: 0.2303 Epoch 17/50 660/660 [==============================] - 2s 4ms/step - loss: 2.2004 - acc: 0.4136 - val_loss: 2.4873 - val_acc: 0.2242 Epoch 18/50 660/660 [==============================] - 3s 4ms/step - loss: 2.1718 - acc: 0.3955 - val_loss: 2.4878 - val_acc: 0.2485 Epoch 19/50 660/660 [==============================] - 3s 4ms/step - loss: 2.1443 - acc: 0.4091 - val_loss: 2.4737 - val_acc: 0.2121 Epoch 20/50 660/660 [==============================] - 3s 4ms/step - loss: 2.1206 - acc: 0.4045 - val_loss: 2.4789 - val_acc: 0.1939 Epoch 21/50 660/660 [==============================] - 2s 4ms/step - loss: 2.0936 - acc: 0.4106 - val_loss: 2.4885 - val_acc: 0.2061 Epoch 22/50 660/660 [==============================] - 3s 4ms/step - loss: 2.0695 - acc: 0.3909 - val_loss: 2.4734 - val_acc: 0.2061 Epoch 23/50 660/660 [==============================] - 3s 4ms/step - loss: 2.0459 - acc: 0.4015 - val_loss: 2.4719 - val_acc: 0.2182 Epoch 24/50 660/660 [==============================] - 3s 4ms/step - loss: 2.0226 - acc: 0.3955 - val_loss: 2.4512 - val_acc: 0.2061 Epoch 25/50 660/660 [==============================] - 3s 4ms/step - loss: 1.9992 - acc: 0.4167 - val_loss: 2.4452 - val_acc: 0.2061 Epoch 26/50 660/660 [==============================] - 3s 5ms/step - loss: 1.9742 - acc: 0.4030 - val_loss: 2.4538 - val_acc: 0.2061 Epoch 27/50 660/660 [==============================] - 3s 4ms/step - loss: 1.9408 - acc: 0.4318 - val_loss: 2.4473 - val_acc: 0.2182 Epoch 28/50 660/660 [==============================] - 3s 4ms/step - loss: 1.9170 - acc: 0.4348 - val_loss: 2.4474 - val_acc: 0.2182 Epoch 29/50 660/660 [==============================] - 3s 4ms/step - loss: 1.8932 - acc: 0.4318 - val_loss: 2.4421 - val_acc: 0.2303 Epoch 30/50 660/660 [==============================] - 3s 4ms/step - loss: 1.8775 - acc: 0.4288 - val_loss: 2.4538 - val_acc: 0.2364 Epoch 31/50 660/660 [==============================] - 3s 4ms/step - loss: 1.8494 - acc: 0.4318 - val_loss: 2.4490 - val_acc: 0.2182 Epoch 32/50 660/660 [==============================] - 3s 4ms/step - loss: 1.8294 - acc: 0.4197 - val_loss: 2.4581 - val_acc: 0.2061 Epoch 33/50 660/660 [==============================] - 3s 4ms/step - loss: 1.8165 - acc: 0.4470 - val_loss: 2.4783 - val_acc: 0.2121 Epoch 34/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7953 - acc: 0.4455 - val_loss: 2.4849 - val_acc: 0.2242 Epoch 35/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7764 - acc: 0.4409 - val_loss: 2.4749 - val_acc: 0.2061 Epoch 36/50 660/660 [==============================] - 2s 4ms/step - loss: 1.7495 - acc: 0.4545 - val_loss: 2.4756 - val_acc: 0.2121 Epoch 37/50 660/660 [==============================] - 2s 4ms/step - loss: 1.7375 - acc: 0.4606 - val_loss: 2.4539 - val_acc: 0.2061 Epoch 38/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7201 - acc: 0.4530 - val_loss: 2.4792 - val_acc: 0.2121 Epoch 39/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7031 - acc: 0.4606 - val_loss: 2.4900 - val_acc: 0.2000 Epoch 40/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6905 - acc: 0.4621 - val_loss: 2.4865 - val_acc: 0.2242 Epoch 41/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6801 - acc: 0.4697 - val_loss: 2.4581 - val_acc: 0.2242 Epoch 42/50 660/660 [==============================] - 2s 4ms/step - loss: 1.6617 - acc: 0.4758 - val_loss: 2.4771 - val_acc: 0.2121 Epoch 43/50 660/660 [==============================] - 2s 4ms/step - loss: 1.6476 - acc: 0.4712 - val_loss: 2.5022 - val_acc: 0.2303 Epoch 44/50 660/660 [==============================] - 3s 5ms/step - loss: 1.6421 - acc: 0.4758 - val_loss: 2.4927 - val_acc: 0.2182 Epoch 45/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6230 - acc: 0.4667 - val_loss: 2.5205 - val_acc: 0.2121 Epoch 46/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6138 - acc: 0.4773 - val_loss: 2.5004 - val_acc: 0.1939 Epoch 47/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6027 - acc: 0.4742 - val_loss: 2.4780 - val_acc: 0.2303 Epoch 48/50 660/660 [==============================] - 2s 4ms/step - loss: 1.5851 - acc: 0.4667 - val_loss: 2.4746 - val_acc: 0.2182 Epoch 49/50 660/660 [==============================] - 3s 4ms/step - loss: 1.5811 - acc: 0.4545 - val_loss: 2.5076 - val_acc: 0.2242 Epoch 50/50 660/660 [==============================] - 3s 4ms/step - loss: 1.5604 - acc: 0.4576 - val_loss: 2.5224 - val_acc: 0.2242
The model is overfitting.
We can see that the validation accuracy is consistently hovering in the low 20s, while the training accuracy is increasing almost to 50%. I wonder why this is. Potentially, the extra dense layer.
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=768, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=50, validation_split=0.2, shuffle=False)
Train on 660 samples, validate on 165 samples Epoch 1/50 660/660 [==============================] - 9s 13ms/step - loss: 2.6689 - acc: 0.1394 - val_loss: 2.4248 - val_acc: 0.2364 Epoch 2/50 660/660 [==============================] - 2s 4ms/step - loss: 2.2359 - acc: 0.2939 - val_loss: 2.3608 - val_acc: 0.2667 Epoch 3/50 660/660 [==============================] - 2s 4ms/step - loss: 2.0112 - acc: 0.3848 - val_loss: 2.3277 - val_acc: 0.2727 Epoch 4/50 660/660 [==============================] - 2s 4ms/step - loss: 1.8258 - acc: 0.4636 - val_loss: 2.3499 - val_acc: 0.2667 Epoch 5/50 660/660 [==============================] - 3s 4ms/step - loss: 1.5919 - acc: 0.5576 - val_loss: 2.3075 - val_acc: 0.2848 Epoch 6/50 660/660 [==============================] - 3s 4ms/step - loss: 1.3799 - acc: 0.6470 - val_loss: 2.2804 - val_acc: 0.3030 Epoch 7/50 660/660 [==============================] - 2s 4ms/step - loss: 1.2080 - acc: 0.7152 - val_loss: 2.2780 - val_acc: 0.3152 Epoch 8/50 660/660 [==============================] - 2s 4ms/step - loss: 1.0735 - acc: 0.7606 - val_loss: 2.2857 - val_acc: 0.3273 Epoch 9/50 660/660 [==============================] - 2s 4ms/step - loss: 0.9545 - acc: 0.8030 - val_loss: 2.2960 - val_acc: 0.3455 Epoch 10/50 660/660 [==============================] - 2s 4ms/step - loss: 0.8460 - acc: 0.8424 - val_loss: 2.3136 - val_acc: 0.3212 Epoch 11/50 660/660 [==============================] - 2s 4ms/step - loss: 0.7589 - acc: 0.8697 - val_loss: 2.3357 - val_acc: 0.2970 Epoch 12/50 660/660 [==============================] - 3s 4ms/step - loss: 0.6945 - acc: 0.8803 - val_loss: 2.3695 - val_acc: 0.3030 Epoch 13/50 660/660 [==============================] - 2s 4ms/step - loss: 0.6406 - acc: 0.9030 - val_loss: 2.4448 - val_acc: 0.2848 Epoch 14/50 660/660 [==============================] - 2s 4ms/step - loss: 0.5893 - acc: 0.9091 - val_loss: 2.5395 - val_acc: 0.2788 Epoch 15/50 660/660 [==============================] - 3s 4ms/step - loss: 0.5348 - acc: 0.9227 - val_loss: 2.5650 - val_acc: 0.2667 Epoch 16/50 660/660 [==============================] - 2s 4ms/step - loss: 0.4660 - acc: 0.9439 - val_loss: 2.5769 - val_acc: 0.2788 Epoch 17/50 660/660 [==============================] - 3s 4ms/step - loss: 0.4120 - acc: 0.9561 - val_loss: 2.5677 - val_acc: 0.2848 Epoch 18/50 660/660 [==============================] - 3s 4ms/step - loss: 0.3531 - acc: 0.9727 - val_loss: 2.5526 - val_acc: 0.2848 Epoch 19/50 660/660 [==============================] - 3s 4ms/step - loss: 0.3079 - acc: 0.9864 - val_loss: 2.5439 - val_acc: 0.3030 Epoch 20/50 660/660 [==============================] - 2s 4ms/step - loss: 0.2750 - acc: 0.9909 - val_loss: 2.5409 - val_acc: 0.2970 Epoch 21/50 660/660 [==============================] - 2s 4ms/step - loss: 0.2488 - acc: 0.9909 - val_loss: 2.5325 - val_acc: 0.2788 Epoch 22/50 660/660 [==============================] - 2s 4ms/step - loss: 0.2284 - acc: 0.9939 - val_loss: 2.5355 - val_acc: 0.2970 Epoch 23/50 660/660 [==============================] - 2s 4ms/step - loss: 0.2214 - acc: 0.9924 - val_loss: 2.5648 - val_acc: 0.2909 Epoch 24/50 660/660 [==============================] - 3s 4ms/step - loss: 0.1959 - acc: 0.9985 - val_loss: 2.5766 - val_acc: 0.2909 Epoch 25/50 660/660 [==============================] - 3s 4ms/step - loss: 0.1754 - acc: 1.0000 - val_loss: 2.5920 - val_acc: 0.3030 Epoch 26/50 660/660 [==============================] - 3s 4ms/step - loss: 0.1610 - acc: 1.0000 - val_loss: 2.6162 - val_acc: 0.3152 Epoch 27/50 660/660 [==============================] - 2s 4ms/step - loss: 0.1506 - acc: 1.0000 - val_loss: 2.6407 - val_acc: 0.3091 Epoch 28/50 660/660 [==============================] - 3s 4ms/step - loss: 0.1400 - acc: 1.0000 - val_loss: 2.6511 - val_acc: 0.3091 Epoch 29/50 650/660 [============================>.] - ETA: 0s - loss: 0.1300 - acc: 1.0000
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-526-35ee41a6b670> in <module>() 13 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 14 ---> 15 second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=50, validation_split=0.2, shuffle=False) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 231 sample_weights=val_sample_weights, 232 batch_size=batch_size, --> 233 verbose=0) 234 if not isinstance(val_outs, list): 235 val_outs = [val_outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in test_loop(model, inputs, targets, sample_weights, batch_size, verbose, steps) 437 ins_batch[i] = ins_batch[i].toarray() 438 --> 439 batch_outs = f(ins_batch) 440 441 if isinstance(batch_outs, list): C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
This problem is even more serious than I thought... it was able to achieve a 100% accuracy on the training set but only 30% on validation...
I think the main problem is that there are way too many learnable weights in the dense layer. Some downsampling is needed.
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=192, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=50, validation_split=0.2, shuffle=False)
Train on 660 samples, validate on 165 samples Epoch 1/50 660/660 [==============================] - 9s 14ms/step - loss: 2.7445 - acc: 0.0500 - val_loss: 2.7121 - val_acc: 0.0667 Epoch 2/50 660/660 [==============================] - 3s 4ms/step - loss: 2.7324 - acc: 0.0455 - val_loss: 2.7032 - val_acc: 0.0667 Epoch 3/50 660/660 [==============================] - 3s 5ms/step - loss: 2.7165 - acc: 0.0561 - val_loss: 2.6807 - val_acc: 0.1212 Epoch 4/50 660/660 [==============================] - 3s 5ms/step - loss: 2.6852 - acc: 0.0833 - val_loss: 2.6606 - val_acc: 0.1030 Epoch 5/50 660/660 [==============================] - 3s 5ms/step - loss: 2.6207 - acc: 0.1152 - val_loss: 2.5700 - val_acc: 0.1636 Epoch 6/50 660/660 [==============================] - 3s 4ms/step - loss: 2.5406 - acc: 0.1712 - val_loss: 2.4960 - val_acc: 0.1818 Epoch 7/50 660/660 [==============================] - 3s 4ms/step - loss: 2.4577 - acc: 0.2000 - val_loss: 2.4385 - val_acc: 0.2121 Epoch 8/50 660/660 [==============================] - 3s 4ms/step - loss: 2.3825 - acc: 0.2348 - val_loss: 2.3852 - val_acc: 0.2424 Epoch 9/50 660/660 [==============================] - 3s 5ms/step - loss: 2.3152 - acc: 0.2530 - val_loss: 2.3536 - val_acc: 0.2485 Epoch 10/50 660/660 [==============================] - 3s 5ms/step - loss: 2.2626 - acc: 0.2879 - val_loss: 2.3401 - val_acc: 0.2424 Epoch 11/50 660/660 [==============================] - 3s 4ms/step - loss: 2.2151 - acc: 0.3015 - val_loss: 2.3869 - val_acc: 0.2485 Epoch 12/50 660/660 [==============================] - 3s 4ms/step - loss: 2.1730 - acc: 0.3197 - val_loss: 2.3695 - val_acc: 0.2364 Epoch 13/50 660/660 [==============================] - 3s 4ms/step - loss: 2.1328 - acc: 0.3318 - val_loss: 2.3196 - val_acc: 0.2667 Epoch 14/50 660/660 [==============================] - 3s 4ms/step - loss: 2.0719 - acc: 0.3545 - val_loss: 2.3011 - val_acc: 0.2727 Epoch 15/50 660/660 [==============================] - 3s 5ms/step - loss: 2.0212 - acc: 0.3712 - val_loss: 2.3069 - val_acc: 0.2727 Epoch 16/50 660/660 [==============================] - 3s 4ms/step - loss: 1.9761 - acc: 0.3924 - val_loss: 2.3171 - val_acc: 0.2485 Epoch 17/50 660/660 [==============================] - 3s 4ms/step - loss: 1.9414 - acc: 0.4000 - val_loss: 2.3017 - val_acc: 0.2606 Epoch 18/50 660/660 [==============================] - 3s 4ms/step - loss: 1.9081 - acc: 0.4136 - val_loss: 2.3024 - val_acc: 0.2545 Epoch 19/50 660/660 [==============================] - 3s 5ms/step - loss: 1.8704 - acc: 0.4182 - val_loss: 2.2894 - val_acc: 0.2606 Epoch 20/50 660/660 [==============================] - 3s 5ms/step - loss: 1.8670 - acc: 0.4136 - val_loss: 2.2733 - val_acc: 0.2848 Epoch 21/50 660/660 [==============================] - 3s 5ms/step - loss: 1.8213 - acc: 0.4394 - val_loss: 2.2551 - val_acc: 0.2909 Epoch 22/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7908 - acc: 0.4515 - val_loss: 2.2363 - val_acc: 0.2909 Epoch 23/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7533 - acc: 0.4773 - val_loss: 2.2280 - val_acc: 0.2909 Epoch 24/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7297 - acc: 0.4636 - val_loss: 2.2292 - val_acc: 0.2909 Epoch 25/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6872 - acc: 0.4894 - val_loss: 2.2218 - val_acc: 0.3091 Epoch 26/50 660/660 [==============================] - 3s 5ms/step - loss: 1.6658 - acc: 0.4894 - val_loss: 2.2267 - val_acc: 0.3091 Epoch 27/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6416 - acc: 0.5091 - val_loss: 2.2227 - val_acc: 0.3212 Epoch 28/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6146 - acc: 0.5091 - val_loss: 2.2258 - val_acc: 0.3091 Epoch 29/50 660/660 [==============================] - 3s 4ms/step - loss: 1.5695 - acc: 0.5394 - val_loss: 2.2055 - val_acc: 0.3455 Epoch 30/50 660/660 [==============================] - 3s 4ms/step - loss: 1.5468 - acc: 0.5424 - val_loss: 2.2128 - val_acc: 0.3152 Epoch 31/50 660/660 [==============================] - 3s 5ms/step - loss: 1.5132 - acc: 0.5636 - val_loss: 2.2191 - val_acc: 0.3394 Epoch 32/50 660/660 [==============================] - 3s 4ms/step - loss: 1.4833 - acc: 0.5848 - val_loss: 2.2280 - val_acc: 0.3455 Epoch 33/50 660/660 [==============================] - 3s 5ms/step - loss: 1.4694 - acc: 0.5848 - val_loss: 2.2267 - val_acc: 0.3273 Epoch 34/50 660/660 [==============================] - 3s 5ms/step - loss: 1.4336 - acc: 0.5894 - val_loss: 2.2304 - val_acc: 0.3455 Epoch 35/50 660/660 [==============================] - 3s 5ms/step - loss: 1.4012 - acc: 0.6152 - val_loss: 2.2344 - val_acc: 0.3455 Epoch 36/50 660/660 [==============================] - 3s 5ms/step - loss: 1.3844 - acc: 0.6288 - val_loss: 2.2446 - val_acc: 0.3455 Epoch 37/50 660/660 [==============================] - 3s 5ms/step - loss: 1.3552 - acc: 0.6394 - val_loss: 2.2516 - val_acc: 0.3394 Epoch 38/50 660/660 [==============================] - 3s 5ms/step - loss: 1.3249 - acc: 0.6424 - val_loss: 2.2427 - val_acc: 0.3455 Epoch 39/50 660/660 [==============================] - 3s 5ms/step - loss: 1.3102 - acc: 0.6500 - val_loss: 2.2446 - val_acc: 0.3394 Epoch 40/50 660/660 [==============================] - 3s 4ms/step - loss: 1.2837 - acc: 0.6636 - val_loss: 2.2553 - val_acc: 0.3394 Epoch 41/50 660/660 [==============================] - 3s 5ms/step - loss: 1.2586 - acc: 0.6667 - val_loss: 2.2497 - val_acc: 0.3333 Epoch 42/50 660/660 [==============================] - 3s 5ms/step - loss: 1.2433 - acc: 0.6758 - val_loss: 2.2548 - val_acc: 0.3636 Epoch 43/50 660/660 [==============================] - 3s 4ms/step - loss: 1.2165 - acc: 0.6773 - val_loss: 2.2583 - val_acc: 0.3455 Epoch 44/50 660/660 [==============================] - 3s 5ms/step - loss: 1.1911 - acc: 0.6985 - val_loss: 2.2656 - val_acc: 0.3515 Epoch 45/50 660/660 [==============================] - 3s 5ms/step - loss: 1.1637 - acc: 0.7076 - val_loss: 2.2703 - val_acc: 0.3333 Epoch 46/50 660/660 [==============================] - 3s 5ms/step - loss: 1.1413 - acc: 0.7136 - val_loss: 2.2666 - val_acc: 0.3515 Epoch 47/50 660/660 [==============================] - 3s 5ms/step - loss: 1.1217 - acc: 0.7182 - val_loss: 2.2701 - val_acc: 0.3515 Epoch 48/50 660/660 [==============================] - 3s 4ms/step - loss: 1.0971 - acc: 0.7303 - val_loss: 2.2883 - val_acc: 0.3576 Epoch 49/50 660/660 [==============================] - 3s 4ms/step - loss: 1.0738 - acc: 0.7424 - val_loss: 2.2946 - val_acc: 0.3394 Epoch 50/50 660/660 [==============================] - 3s 4ms/step - loss: 1.0494 - acc: 0.7470 - val_loss: 2.2915 - val_acc: 0.3455
It's a step in the right direction. But it's still pretty serious overfitting. I'm not really sure why it's doing that, especially given that I shuffled the training and validation data.
Let's try adding a small dense layer before the first one.
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=192, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=10, epochs=50, validation_split=0.2, shuffle=False)
Train on 660 samples, validate on 165 samples Epoch 1/50 660/660 [==============================] - 10s 15ms/step - loss: 2.7467 - acc: 0.0470 - val_loss: 2.7195 - val_acc: 0.0727 Epoch 2/50 660/660 [==============================] - 3s 4ms/step - loss: 2.7364 - acc: 0.0394 - val_loss: 2.7104 - val_acc: 0.0606 Epoch 3/50 660/660 [==============================] - 3s 4ms/step - loss: 2.7257 - acc: 0.0530 - val_loss: 2.6993 - val_acc: 0.0667 Epoch 4/50 660/660 [==============================] - 3s 5ms/step - loss: 2.7136 - acc: 0.0712 - val_loss: 2.6913 - val_acc: 0.0727 Epoch 5/50 660/660 [==============================] - 3s 5ms/step - loss: 2.6729 - acc: 0.1076 - val_loss: 2.6319 - val_acc: 0.1394 Epoch 6/50 660/660 [==============================] - 3s 4ms/step - loss: 2.5898 - acc: 0.1591 - val_loss: 2.5354 - val_acc: 0.1879 Epoch 7/50 660/660 [==============================] - 3s 4ms/step - loss: 2.4721 - acc: 0.2167 - val_loss: 2.4406 - val_acc: 0.2182 Epoch 8/50 660/660 [==============================] - 3s 5ms/step - loss: 2.3765 - acc: 0.2424 - val_loss: 2.3938 - val_acc: 0.2242 Epoch 9/50 660/660 [==============================] - 3s 4ms/step - loss: 2.2867 - acc: 0.2742 - val_loss: 2.3366 - val_acc: 0.2424 Epoch 10/50 660/660 [==============================] - 3s 5ms/step - loss: 2.2202 - acc: 0.3091 - val_loss: 2.3030 - val_acc: 0.2606 Epoch 11/50 660/660 [==============================] - 3s 5ms/step - loss: 2.1767 - acc: 0.2970 - val_loss: 2.2892 - val_acc: 0.2727 Epoch 12/50 660/660 [==============================] - 3s 4ms/step - loss: 2.1295 - acc: 0.3076 - val_loss: 2.2846 - val_acc: 0.2727 Epoch 13/50 660/660 [==============================] - 3s 4ms/step - loss: 2.0875 - acc: 0.3379 - val_loss: 2.2877 - val_acc: 0.2545 Epoch 14/50 660/660 [==============================] - 3s 4ms/step - loss: 2.0388 - acc: 0.3455 - val_loss: 2.2875 - val_acc: 0.2485 Epoch 15/50 660/660 [==============================] - 3s 5ms/step - loss: 1.9969 - acc: 0.3606 - val_loss: 2.2687 - val_acc: 0.2545 Epoch 16/50 660/660 [==============================] - 3s 5ms/step - loss: 1.9669 - acc: 0.3803 - val_loss: 2.2607 - val_acc: 0.2788 Epoch 17/50 660/660 [==============================] - 3s 4ms/step - loss: 1.9218 - acc: 0.3894 - val_loss: 2.2442 - val_acc: 0.2848 Epoch 18/50 660/660 [==============================] - 3s 5ms/step - loss: 1.9152 - acc: 0.3955 - val_loss: 2.2398 - val_acc: 0.2970 Epoch 19/50 660/660 [==============================] - 3s 4ms/step - loss: 1.8692 - acc: 0.4167 - val_loss: 2.2495 - val_acc: 0.2848 Epoch 20/50 660/660 [==============================] - 3s 4ms/step - loss: 1.8427 - acc: 0.4303 - val_loss: 2.2367 - val_acc: 0.3152 Epoch 21/50 660/660 [==============================] - 3s 5ms/step - loss: 1.8076 - acc: 0.4409 - val_loss: 2.2310 - val_acc: 0.3030 Epoch 22/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7792 - acc: 0.4545 - val_loss: 2.2287 - val_acc: 0.3152 Epoch 23/50 660/660 [==============================] - 3s 4ms/step - loss: 1.7447 - acc: 0.4697 - val_loss: 2.2258 - val_acc: 0.3152 Epoch 24/50 660/660 [==============================] - 3s 5ms/step - loss: 1.7083 - acc: 0.4833 - val_loss: 2.2275 - val_acc: 0.3212 Epoch 25/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6905 - acc: 0.4803 - val_loss: 2.2314 - val_acc: 0.3333 Epoch 26/50 660/660 [==============================] - 3s 5ms/step - loss: 1.6516 - acc: 0.4955 - val_loss: 2.2252 - val_acc: 0.3273 Epoch 27/50 660/660 [==============================] - 3s 5ms/step - loss: 1.6308 - acc: 0.4985 - val_loss: 2.2272 - val_acc: 0.3152 Epoch 28/50 660/660 [==============================] - 3s 4ms/step - loss: 1.6255 - acc: 0.5106 - val_loss: 2.2232 - val_acc: 0.3212 Epoch 29/50 660/660 [==============================] - 3s 5ms/step - loss: 1.5772 - acc: 0.5288 - val_loss: 2.2140 - val_acc: 0.3333 Epoch 30/50 660/660 [==============================] - 3s 5ms/step - loss: 1.5499 - acc: 0.5364 - val_loss: 2.2423 - val_acc: 0.3152 Epoch 31/50 660/660 [==============================] - 3s 5ms/step - loss: 1.5066 - acc: 0.5515 - val_loss: 2.2347 - val_acc: 0.3273 Epoch 32/50 660/660 [==============================] - 3s 5ms/step - loss: 1.4806 - acc: 0.5636 - val_loss: 2.2244 - val_acc: 0.3333 Epoch 33/50 660/660 [==============================] - 3s 4ms/step - loss: 1.4700 - acc: 0.5788 - val_loss: 2.2456 - val_acc: 0.3394 Epoch 34/50 660/660 [==============================] - 3s 5ms/step - loss: 1.4385 - acc: 0.5803 - val_loss: 2.2533 - val_acc: 0.3394 Epoch 35/50 660/660 [==============================] - 3s 4ms/step - loss: 1.3944 - acc: 0.5970 - val_loss: 2.2527 - val_acc: 0.3394 Epoch 36/50 660/660 [==============================] - 3s 5ms/step - loss: 1.3724 - acc: 0.6061 - val_loss: 2.2564 - val_acc: 0.3515 Epoch 37/50 660/660 [==============================] - 3s 5ms/step - loss: 1.3496 - acc: 0.6167 - val_loss: 2.2879 - val_acc: 0.3333 Epoch 38/50 660/660 [==============================] - 3s 4ms/step - loss: 1.3195 - acc: 0.6348 - val_loss: 2.2692 - val_acc: 0.3576 Epoch 39/50 660/660 [==============================] - 3s 4ms/step - loss: 1.2948 - acc: 0.6318 - val_loss: 2.2807 - val_acc: 0.3636 Epoch 40/50 660/660 [==============================] - 3s 5ms/step - loss: 1.2801 - acc: 0.6333 - val_loss: 2.2866 - val_acc: 0.3394 Epoch 41/50 660/660 [==============================] - 3s 5ms/step - loss: 1.2748 - acc: 0.6379 - val_loss: 2.2882 - val_acc: 0.3636 Epoch 42/50 660/660 [==============================] - 3s 5ms/step - loss: 1.2317 - acc: 0.6591 - val_loss: 2.2910 - val_acc: 0.3515 Epoch 43/50 660/660 [==============================] - 3s 4ms/step - loss: 1.2060 - acc: 0.6652 - val_loss: 2.3153 - val_acc: 0.3212 Epoch 44/50 660/660 [==============================] - 3s 5ms/step - loss: 1.1972 - acc: 0.6667 - val_loss: 2.3113 - val_acc: 0.3636 Epoch 45/50 660/660 [==============================] - 3s 4ms/step - loss: 1.1507 - acc: 0.6879 - val_loss: 2.3190 - val_acc: 0.3576 Epoch 46/50 660/660 [==============================] - 3s 4ms/step - loss: 1.1244 - acc: 0.6970 - val_loss: 2.3333 - val_acc: 0.3394 Epoch 47/50 660/660 [==============================] - 3s 5ms/step - loss: 1.1498 - acc: 0.6803 - val_loss: 2.3707 - val_acc: 0.3273 Epoch 48/50 660/660 [==============================] - 3s 5ms/step - loss: 1.1242 - acc: 0.7000 - val_loss: 2.3361 - val_acc: 0.3636 Epoch 49/50 660/660 [==============================] - 3s 4ms/step - loss: 1.0785 - acc: 0.7303 - val_loss: 2.3488 - val_acc: 0.3394 Epoch 50/50 660/660 [==============================] - 3s 4ms/step - loss: 1.0471 - acc: 0.7348 - val_loss: 2.3602 - val_acc: 0.3576
It looks like a step in the right direction. The addition of a small dense layer before the output layer has reduced the number of learnable parameters, which tends to lead to a reduction in overfitting as well. But now it's failing to fit altogether.
It is worth thinking about the fully-connected-layer architecture at this point. The ReLU + softmax + max-pool + softmax + max-pool architecture of the convolutional/pooling part appears to have succeeded, with relatively minimal parameters. But the dense layers also form a crucial part of the CNN architecture.
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=192, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=10, epochs=50, validation_split=0.2)
Train on 30000 samples, validate on 7500 samples Epoch 1/50 30000/30000 [==============================] - 154s 5ms/step - loss: 2.5256 - acc: 0.1672 - val_loss: 2.3508 - val_acc: 0.2384 Epoch 2/50 30000/30000 [==============================] - 142s 5ms/step - loss: 2.2412 - acc: 0.2753 - val_loss: 2.1693 - val_acc: 0.3096 Epoch 3/50 30000/30000 [==============================] - 141s 5ms/step - loss: 2.1529 - acc: 0.3078 - val_loss: 2.1442 - val_acc: 0.3124 Epoch 4/50 30000/30000 [==============================] - 141s 5ms/step - loss: 2.1169 - acc: 0.3196 - val_loss: 2.1186 - val_acc: 0.3204 Epoch 5/50 30000/30000 [==============================] - 141s 5ms/step - loss: 2.0830 - acc: 0.3313 - val_loss: 2.0968 - val_acc: 0.3228 Epoch 6/50 30000/30000 [==============================] - 139s 5ms/step - loss: 2.0678 - acc: 0.3336 - val_loss: 2.0761 - val_acc: 0.3339 Epoch 7/50 30000/30000 [==============================] - 139s 5ms/step - loss: 2.0524 - acc: 0.3386 - val_loss: 2.0766 - val_acc: 0.3321 Epoch 8/50 30000/30000 [==============================] - 139s 5ms/step - loss: 2.0408 - acc: 0.3431 - val_loss: 2.0689 - val_acc: 0.3291 Epoch 9/50 30000/30000 [==============================] - 143s 5ms/step - loss: 2.0223 - acc: 0.3505 - val_loss: 2.0780 - val_acc: 0.3216 Epoch 10/50 30000/30000 [==============================] - 148s 5ms/step - loss: 2.0179 - acc: 0.3532 - val_loss: 2.0554 - val_acc: 0.3321 Epoch 11/50 30000/30000 [==============================] - 140s 5ms/step - loss: 2.0098 - acc: 0.3564 - val_loss: 2.0573 - val_acc: 0.3304 Epoch 12/50 30000/30000 [==============================] - 143s 5ms/step - loss: 1.9958 - acc: 0.3598 - val_loss: 2.0488 - val_acc: 0.3451 Epoch 13/50 30000/30000 [==============================] - 139s 5ms/step - loss: 1.9928 - acc: 0.3595 - val_loss: 2.0227 - val_acc: 0.3521 Epoch 14/50 30000/30000 [==============================] - 147s 5ms/step - loss: 1.9778 - acc: 0.3680 - val_loss: 2.0353 - val_acc: 0.3429 Epoch 15/50 30000/30000 [==============================] - 147s 5ms/step - loss: 1.9887 - acc: 0.3606 - val_loss: 2.0350 - val_acc: 0.3443 Epoch 16/50 30000/30000 [==============================] - 146s 5ms/step - loss: 1.9690 - acc: 0.3670 - val_loss: 2.0102 - val_acc: 0.3459 Epoch 17/50 30000/30000 [==============================] - 148s 5ms/step - loss: 1.9628 - acc: 0.3707 - val_loss: 2.0379 - val_acc: 0.3464 Epoch 18/50 30000/30000 [==============================] - 146s 5ms/step - loss: 1.9656 - acc: 0.3682 - val_loss: 2.0355 - val_acc: 0.3480 Epoch 19/50 30000/30000 [==============================] - 146s 5ms/step - loss: 1.9575 - acc: 0.3723 - val_loss: 2.0325 - val_acc: 0.3461 Epoch 20/50 30000/30000 [==============================] - 144s 5ms/step - loss: 1.9572 - acc: 0.3713 - val_loss: 1.9908 - val_acc: 0.3559 Epoch 21/50 30000/30000 [==============================] - 142s 5ms/step - loss: 1.9512 - acc: 0.3746 - val_loss: 2.0204 - val_acc: 0.3455 Epoch 22/50 30000/30000 [==============================] - 139s 5ms/step - loss: 1.9594 - acc: 0.3704 - val_loss: 2.0018 - val_acc: 0.3464 Epoch 23/50 30000/30000 [==============================] - 143s 5ms/step - loss: 1.9498 - acc: 0.3766 - val_loss: 1.9810 - val_acc: 0.3652 Epoch 24/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9413 - acc: 0.3784 - val_loss: 1.9958 - val_acc: 0.3563 Epoch 25/50 30000/30000 [==============================] - 144s 5ms/step - loss: 1.9371 - acc: 0.3780 - val_loss: 1.9952 - val_acc: 0.3565 Epoch 26/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9347 - acc: 0.3772 - val_loss: 1.9847 - val_acc: 0.3616 Epoch 27/50 30000/30000 [==============================] - 142s 5ms/step - loss: 1.9314 - acc: 0.3808 - val_loss: 1.9937 - val_acc: 0.3649 Epoch 28/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9303 - acc: 0.3786 - val_loss: 2.0085 - val_acc: 0.3500 Epoch 29/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9255 - acc: 0.3802 - val_loss: 1.9791 - val_acc: 0.3611 Epoch 30/50 30000/30000 [==============================] - 139s 5ms/step - loss: 1.9275 - acc: 0.3814 - val_loss: 2.0043 - val_acc: 0.3548 Epoch 31/50 30000/30000 [==============================] - 139s 5ms/step - loss: 1.9356 - acc: 0.3782 - val_loss: 1.9993 - val_acc: 0.3605 Epoch 32/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9209 - acc: 0.3834 - val_loss: 1.9886 - val_acc: 0.3623 Epoch 33/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9151 - acc: 0.3846 - val_loss: 1.9743 - val_acc: 0.3684 Epoch 34/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9356 - acc: 0.3805 - val_loss: 2.0514 - val_acc: 0.3436 Epoch 35/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9168 - acc: 0.3828 - val_loss: 1.9626 - val_acc: 0.3693 Epoch 36/50 30000/30000 [==============================] - 150s 5ms/step - loss: 1.9102 - acc: 0.3866 - val_loss: 1.9824 - val_acc: 0.3637 Epoch 37/50 30000/30000 [==============================] - 146s 5ms/step - loss: 1.9240 - acc: 0.3813 - val_loss: 1.9521 - val_acc: 0.3757 Epoch 38/50 30000/30000 [==============================] - 140s 5ms/step - loss: 1.9166 - acc: 0.3830 - val_loss: 1.9706 - val_acc: 0.3659 Epoch 39/50 30000/30000 [==============================] - 147s 5ms/step - loss: 1.9090 - acc: 0.3832 - val_loss: 1.9831 - val_acc: 0.3663 Epoch 40/50 30000/30000 [==============================] - 147s 5ms/step - loss: 1.9150 - acc: 0.3841 - val_loss: 1.9905 - val_acc: 0.3577 Epoch 41/50 30000/30000 [==============================] - 141s 5ms/step - loss: 1.9061 - acc: 0.3858 - val_loss: 1.9681 - val_acc: 0.3687 Epoch 42/50 30000/30000 [==============================] - 145s 5ms/step - loss: 1.9107 - acc: 0.3880 - val_loss: 1.9595 - val_acc: 0.3700 Epoch 43/50 30000/30000 [==============================] - 143s 5ms/step - loss: 1.9111 - acc: 0.3825 - val_loss: 1.9569 - val_acc: 0.3709 Epoch 44/50 30000/30000 [==============================] - 141s 5ms/step - loss: 1.9136 - acc: 0.3863 - val_loss: 1.9771 - val_acc: 0.3668 Epoch 45/50 30000/30000 [==============================] - 142s 5ms/step - loss: 1.9159 - acc: 0.3833 - val_loss: 1.9834 - val_acc: 0.3619 Epoch 46/50 30000/30000 [==============================] - 142s 5ms/step - loss: 1.9017 - acc: 0.3861 - val_loss: 1.9552 - val_acc: 0.3751 Epoch 47/50 30000/30000 [==============================] - 141s 5ms/step - loss: 1.9087 - acc: 0.3858 - val_loss: 2.0149 - val_acc: 0.3577 Epoch 48/50 30000/30000 [==============================] - 142s 5ms/step - loss: 1.9392 - acc: 0.3759 - val_loss: 1.9731 - val_acc: 0.3639 Epoch 49/50 30000/30000 [==============================] - 142s 5ms/step - loss: 1.9084 - acc: 0.3865 - val_loss: 1.9862 - val_acc: 0.3657 Epoch 50/50 30000/30000 [==============================] - 141s 5ms/step - loss: 1.9288 - acc: 0.3805 - val_loss: 1.9416 - val_acc: 0.3833
I have redefined the "large" dataset of 45000 samples to achieve class balance. With class imbalance I trained on the most recent setup and achieved 20%. With class balance I achieved 38% accuracy. This is a much better result suggesting that the reason behind most of our previous struggles was probably class imbalance. But it's still not great because it looks like the neural network is struggling to get past 38% accuracy-wise, although surprisingly enough this accuracy is mirrored in the validation set and it's already higher than anything we've had with the white images.
It's probably a signal that the low number of images in the white dataset have made it easier for the neural network to more completely memorize the training set, which is obviously not feasible with 30000 samples. But notice also that the neural network appears to have converged to a local minimum with the large dataset which might mean it's struggling to get rid of the background noise. Therefore I will stay with the white dataset for a bit longer.
Regarding the overfitting I am considering tuning the batch size as a hyperparameter. Perhaps the low batch size made it easier for the network to overfit.
search_space = [2, 4, 8, 16, 32, 64, 128][::-1]
for s in search_space:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=192, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=s, epochs=35, validation_split=0.2, shuffle=False, verbose=False)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size {s}:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Batch size 128: 26.5% (train) vs. 24.200000000000003% (validation) Batch size 64: 39.900000000000006% (train) vs. 29.6% (validation) Batch size 32: 42.5% (train) vs. 32.1% (validation) Batch size 16: 54.6% (train) vs. 25.400000000000002% (validation) Batch size 8: 64.5% (train) vs. 33.9% (validation) Batch size 4: 70.10000000000001% (train) vs. 30.900000000000002% (validation) Batch size 2: 71.5% (train) vs. 26.6% (validation)
There are drastic amounts of overfitting going on with batch sizes of less than or equal to 16. It looks like a batch size of 32 achieves the least amount of overfitting while actually fitting to the training set, and this is what many sources will state. But it's nice to see it in practice.
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=192, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False, verbose=False)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Batch size 32: 58.0% (train) vs. 34.5% (validation)
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=192, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False, verbose=False)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Batch size 32: 54.800000000000004% (train) vs. 32.7% (validation)
Date: 20 November
I have just realized that potentially it is the high number of epochs that is causing the overfitting. 50 cycles of the training set is way too many and it is enough to allow the model to memorize the training set. Trying with only 25 epochs instead, but with the same setup.
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same'))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=15, input_dim=192, activation='softmax'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=25, validation_split=0.2, shuffle=False, verbose=False)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Batch size 32: 42.2% (train) vs. 30.3% (validation)
Less overfitting this time. So now the main task is to improve the accuracy, and this can be done by reworking the architecture of the neural network.
The convolutional + max-pooling part already consists of multiple layers. But now it's the dense part that should be improved. The dense layers, particularly the hidden layers, play an important role in using the features extracted by the convolutional layers to make deductions. One layer is probably not good enough. Therefore I will place another layer before it. It is important that this layer not have too few or too many neurons, 20 seems like a good number.
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=192, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=25, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 660 samples, validate on 165 samples Epoch 1/25 660/660 [==============================] - 17s 25ms/step - loss: 2.7090 - acc: 0.0515 - val_loss: 2.7081 - val_acc: 0.0667 Epoch 2/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7084 - acc: 0.0606 - val_loss: 2.7080 - val_acc: 0.0667 Epoch 3/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7083 - acc: 0.0530 - val_loss: 2.7078 - val_acc: 0.0727 Epoch 4/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7077 - acc: 0.0667 - val_loss: 2.7072 - val_acc: 0.0667 Epoch 5/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7057 - acc: 0.0636 - val_loss: 2.7052 - val_acc: 0.0727 Epoch 6/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7007 - acc: 0.0970 - val_loss: 2.6971 - val_acc: 0.1273 Epoch 7/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6888 - acc: 0.1136 - val_loss: 2.6853 - val_acc: 0.1091 Epoch 8/25 660/660 [==============================] - 3s 4ms/step - loss: 2.6769 - acc: 0.1121 - val_loss: 2.6784 - val_acc: 0.1212 Epoch 9/25 660/660 [==============================] - 2s 4ms/step - loss: 2.6675 - acc: 0.1136 - val_loss: 2.6702 - val_acc: 0.1212 Epoch 10/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6603 - acc: 0.1182 - val_loss: 2.6669 - val_acc: 0.1212 Epoch 11/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6536 - acc: 0.1121 - val_loss: 2.6651 - val_acc: 0.1212 Epoch 12/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6468 - acc: 0.1136 - val_loss: 2.6579 - val_acc: 0.1455 Epoch 13/25 660/660 [==============================] - 2s 4ms/step - loss: 2.6448 - acc: 0.1121 - val_loss: 2.6660 - val_acc: 0.1030 Epoch 14/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6413 - acc: 0.1318 - val_loss: 2.6662 - val_acc: 0.1030 Epoch 15/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6440 - acc: 0.1227 - val_loss: 2.6455 - val_acc: 0.1333 Epoch 16/25 660/660 [==============================] - 2s 4ms/step - loss: 2.6373 - acc: 0.1212 - val_loss: 2.6364 - val_acc: 0.1333 Epoch 17/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6276 - acc: 0.1394 - val_loss: 2.6375 - val_acc: 0.1333 Epoch 18/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6222 - acc: 0.1348 - val_loss: 2.6413 - val_acc: 0.1212 Epoch 19/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6204 - acc: 0.1303 - val_loss: 2.6416 - val_acc: 0.1212 Epoch 20/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6161 - acc: 0.1318 - val_loss: 2.6395 - val_acc: 0.1273 Epoch 21/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6113 - acc: 0.1318 - val_loss: 2.6370 - val_acc: 0.1394 Epoch 22/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6075 - acc: 0.1318 - val_loss: 2.6354 - val_acc: 0.1394 Epoch 23/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6037 - acc: 0.1318 - val_loss: 2.6335 - val_acc: 0.1333 Epoch 24/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6002 - acc: 0.1333 - val_loss: 2.6317 - val_acc: 0.1333 Epoch 25/25 660/660 [==============================] - 2s 3ms/step - loss: 2.5967 - acc: 0.1333 - val_loss: 2.6302 - val_acc: 0.1273 Batch size 32: 12.8% (train) vs. 12.700000000000001% (validation)
It got a lot worse.
Quick update: I added initializers and biases to every layer. After this change, when I use 1 dense layer it gets 40% train vs. 30% validation - on par with before.
Adding an additional dense layer brings it down to 13% vs. 13% which is horrendous. Perhaps I should be using ReLU in the first dense layer.
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='softmax', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=25, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 660 samples, validate on 165 samples Epoch 1/25 660/660 [==============================] - 16s 24ms/step - loss: 2.7104 - acc: 0.0621 - val_loss: 2.7081 - val_acc: 0.0667 Epoch 2/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7087 - acc: 0.0545 - val_loss: 2.7081 - val_acc: 0.0667 Epoch 3/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7084 - acc: 0.0667 - val_loss: 2.7077 - val_acc: 0.1091 Epoch 4/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7080 - acc: 0.0727 - val_loss: 2.7075 - val_acc: 0.0788 Epoch 5/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7068 - acc: 0.0955 - val_loss: 2.7055 - val_acc: 0.1030 Epoch 6/25 660/660 [==============================] - 2s 3ms/step - loss: 2.7037 - acc: 0.0970 - val_loss: 2.7004 - val_acc: 0.1212 Epoch 7/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6915 - acc: 0.1030 - val_loss: 2.6801 - val_acc: 0.1030 Epoch 8/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6593 - acc: 0.1121 - val_loss: 2.6476 - val_acc: 0.1030 Epoch 9/25 660/660 [==============================] - 2s 3ms/step - loss: 2.6144 - acc: 0.1121 - val_loss: 2.6125 - val_acc: 0.1212 Epoch 10/25 660/660 [==============================] - 2s 3ms/step - loss: 2.5776 - acc: 0.1348 - val_loss: 2.5883 - val_acc: 0.1879 Epoch 11/25 660/660 [==============================] - 2s 3ms/step - loss: 2.5419 - acc: 0.1576 - val_loss: 2.5755 - val_acc: 0.2000 Epoch 12/25 660/660 [==============================] - 2s 3ms/step - loss: 2.5193 - acc: 0.1833 - val_loss: 2.5512 - val_acc: 0.1939 Epoch 13/25 660/660 [==============================] - 2s 3ms/step - loss: 2.4916 - acc: 0.1939 - val_loss: 2.5428 - val_acc: 0.2000 Epoch 14/25 660/660 [==============================] - 2s 3ms/step - loss: 2.4696 - acc: 0.2076 - val_loss: 2.5239 - val_acc: 0.1939 Epoch 15/25 660/660 [==============================] - 2s 3ms/step - loss: 2.4459 - acc: 0.2030 - val_loss: 2.5152 - val_acc: 0.2061 Epoch 16/25 660/660 [==============================] - 2s 3ms/step - loss: 2.4285 - acc: 0.2152 - val_loss: 2.5173 - val_acc: 0.2121 Epoch 17/25 660/660 [==============================] - 2s 3ms/step - loss: 2.4164 - acc: 0.2227 - val_loss: 2.5104 - val_acc: 0.2182 Epoch 18/25 660/660 [==============================] - 2s 3ms/step - loss: 2.3972 - acc: 0.2303 - val_loss: 2.4916 - val_acc: 0.2061 Epoch 19/25 660/660 [==============================] - 2s 3ms/step - loss: 2.4105 - acc: 0.2303 - val_loss: 2.5063 - val_acc: 0.2182 Epoch 20/25 660/660 [==============================] - 2s 3ms/step - loss: 2.3742 - acc: 0.2333 - val_loss: 2.5026 - val_acc: 0.2242 Epoch 21/25 660/660 [==============================] - 2s 3ms/step - loss: 2.3524 - acc: 0.2379 - val_loss: 2.5002 - val_acc: 0.2182 Epoch 22/25 660/660 [==============================] - 2s 3ms/step - loss: 2.3351 - acc: 0.2379 - val_loss: 2.4942 - val_acc: 0.2182 Epoch 23/25 660/660 [==============================] - 2s 3ms/step - loss: 2.3177 - acc: 0.2485 - val_loss: 2.4916 - val_acc: 0.2303 Epoch 24/25 660/660 [==============================] - 2s 3ms/step - loss: 2.3023 - acc: 0.2485 - val_loss: 2.4857 - val_acc: 0.2424 Epoch 25/25 660/660 [==============================] - 2s 3ms/step - loss: 2.2861 - acc: 0.2606 - val_loss: 2.4825 - val_acc: 0.2364 Batch size 32: 25.6% (train) vs. 23.6% (validation)
Improvement.
(I turned on "verbose" so I could see the progressive learning better.)
The accuracy climbs to 26% vs. 24% with this neural network. Apologies for the lack of visual communication.
It's probably for similar reasons to using ReLU in the first convolution.
What if we try similar things with the dense layers to what we did for the convolution/max-pooling? Namely, a hidden layer with softmax, with something like ~35 neurons?
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=48, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=25, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 660 samples, validate on 165 samples Epoch 1/25 660/660 [==============================] - 29s 44ms/step - loss: 2.7086 - acc: 0.0803 - val_loss: 2.7012 - val_acc: 0.0727 Epoch 2/25 660/660 [==============================] - 3s 4ms/step - loss: 2.6896 - acc: 0.0924 - val_loss: 2.6701 - val_acc: 0.0909 Epoch 3/25 660/660 [==============================] - 3s 4ms/step - loss: 2.6032 - acc: 0.1091 - val_loss: 2.5758 - val_acc: 0.1273 Epoch 4/25 660/660 [==============================] - 3s 4ms/step - loss: 2.4927 - acc: 0.1667 - val_loss: 2.5201 - val_acc: 0.1939 Epoch 5/25 660/660 [==============================] - 3s 4ms/step - loss: 2.3998 - acc: 0.2379 - val_loss: 2.4871 - val_acc: 0.1879 Epoch 6/25 660/660 [==============================] - 3s 4ms/step - loss: 2.3114 - acc: 0.2606 - val_loss: 2.4482 - val_acc: 0.2364 Epoch 7/25 660/660 [==============================] - 3s 4ms/step - loss: 2.2420 - acc: 0.2985 - val_loss: 2.4256 - val_acc: 0.2545 Epoch 8/25 660/660 [==============================] - 3s 4ms/step - loss: 2.1868 - acc: 0.3045 - val_loss: 2.4018 - val_acc: 0.2485 Epoch 9/25 660/660 [==============================] - 3s 4ms/step - loss: 2.1354 - acc: 0.3091 - val_loss: 2.4036 - val_acc: 0.2303 Epoch 10/25 660/660 [==============================] - 3s 4ms/step - loss: 2.0890 - acc: 0.3227 - val_loss: 2.3905 - val_acc: 0.2303 Epoch 11/25 660/660 [==============================] - 3s 4ms/step - loss: 2.0376 - acc: 0.3455 - val_loss: 2.3862 - val_acc: 0.2242 Epoch 12/25 660/660 [==============================] - 3s 4ms/step - loss: 1.9931 - acc: 0.3576 - val_loss: 2.3993 - val_acc: 0.2364 Epoch 13/25 660/660 [==============================] - 3s 4ms/step - loss: 1.9548 - acc: 0.3712 - val_loss: 2.4067 - val_acc: 0.2485 Epoch 14/25 660/660 [==============================] - 3s 4ms/step - loss: 1.9258 - acc: 0.3773 - val_loss: 2.4176 - val_acc: 0.2242 Epoch 15/25 660/660 [==============================] - 3s 4ms/step - loss: 1.8956 - acc: 0.3894 - val_loss: 2.4302 - val_acc: 0.2606 Epoch 16/25 660/660 [==============================] - 3s 4ms/step - loss: 1.8659 - acc: 0.3985 - val_loss: 2.4674 - val_acc: 0.2606 Epoch 17/25 660/660 [==============================] - 3s 4ms/step - loss: 1.8402 - acc: 0.4061 - val_loss: 2.5018 - val_acc: 0.2606 Epoch 18/25 660/660 [==============================] - 3s 4ms/step - loss: 1.8208 - acc: 0.4167 - val_loss: 2.5133 - val_acc: 0.2788 Epoch 19/25 660/660 [==============================] - 3s 4ms/step - loss: 1.7969 - acc: 0.4167 - val_loss: 2.4824 - val_acc: 0.2545 Epoch 20/25 660/660 [==============================] - 3s 4ms/step - loss: 1.7749 - acc: 0.4258 - val_loss: 2.4819 - val_acc: 0.2364 Epoch 21/25 660/660 [==============================] - 3s 4ms/step - loss: 1.7432 - acc: 0.4379 - val_loss: 2.4881 - val_acc: 0.2545 Epoch 22/25 660/660 [==============================] - 3s 4ms/step - loss: 1.7069 - acc: 0.4455 - val_loss: 2.5128 - val_acc: 0.2545 Epoch 23/25 660/660 [==============================] - 3s 4ms/step - loss: 1.6788 - acc: 0.4621 - val_loss: 2.5498 - val_acc: 0.2545 Epoch 24/25 660/660 [==============================] - 3s 4ms/step - loss: 1.6525 - acc: 0.4773 - val_loss: 2.5984 - val_acc: 0.2606 Epoch 25/25 640/660 [============================>.] - ETA: 0s - loss: 1.6347 - acc: 0.4797
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-621-db93d9a3aa16> in <module>() 19 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 20 ---> 21 second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=25, validation_split=0.2, shuffle=False) 22 23 tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
So, I had a misconception. I forgot that softmax normalizes outputs to form a probability distribution. But then I realized that it amplifies higher values even further, so it might amplify parts that are actually in the image and not those outside.
Regardless of the fact that I added an extra conv+maxpool to reduce the size of the input to the fully connected part to be 48 instead of 192, overfitting (and also a lack of learning) still occurs which puzzles me.
I wonder whether there are other activation functions that don't normalize to this range [0, 1]. Found nothing, so I changed all activation functions to ReLU except for the last layer.
It still overfits for whatever reason.
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=5, input_dim=48, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=5, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2)
tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(classified_data_2[30000:], one_hot_encode(y_data[30000:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 30000 samples, validate on 7500 samples Epoch 1/25 30000/30000 [==============================] - 150s 5ms/step - loss: 2.5153 - acc: 0.1559 - val_loss: 2.3702 - val_acc: 0.2068 Epoch 2/25 30000/30000 [==============================] - 124s 4ms/step - loss: 2.3563 - acc: 0.2209 - val_loss: 2.3234 - val_acc: 0.2328 Epoch 3/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.2968 - acc: 0.2423 - val_loss: 2.2823 - val_acc: 0.2557 Epoch 4/25 30000/30000 [==============================] - 120s 4ms/step - loss: 2.2561 - acc: 0.2545 - val_loss: 2.2481 - val_acc: 0.2660 Epoch 5/25 30000/30000 [==============================] - 120s 4ms/step - loss: 2.2282 - acc: 0.2658 - val_loss: 2.2241 - val_acc: 0.2772 Epoch 6/25 30000/30000 [==============================] - 118s 4ms/step - loss: 2.2007 - acc: 0.2754 - val_loss: 2.1960 - val_acc: 0.2801 Epoch 7/25 30000/30000 [==============================] - 118s 4ms/step - loss: 2.1896 - acc: 0.2787 - val_loss: 2.1796 - val_acc: 0.2829 Epoch 8/25 30000/30000 [==============================] - 120s 4ms/step - loss: 2.1764 - acc: 0.2825 - val_loss: 2.1613 - val_acc: 0.2903 Epoch 9/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1614 - acc: 0.2908 - val_loss: 2.1444 - val_acc: 0.2977 Epoch 10/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1557 - acc: 0.2929 - val_loss: 2.1598 - val_acc: 0.3071 Epoch 11/25 30000/30000 [==============================] - 120s 4ms/step - loss: 2.1452 - acc: 0.2971 - val_loss: 2.1606 - val_acc: 0.2877 Epoch 12/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1321 - acc: 0.3008 - val_loss: 2.1268 - val_acc: 0.3080 Epoch 13/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1340 - acc: 0.2977 - val_loss: 2.1453 - val_acc: 0.3016 Epoch 14/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1234 - acc: 0.3024 - val_loss: 2.1148 - val_acc: 0.3121 Epoch 15/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1211 - acc: 0.3026 - val_loss: 2.1466 - val_acc: 0.3001 Epoch 16/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1111 - acc: 0.3068 - val_loss: 2.1114 - val_acc: 0.3109 Epoch 17/25 30000/30000 [==============================] - 118s 4ms/step - loss: 2.1118 - acc: 0.3056 - val_loss: 2.1174 - val_acc: 0.3091 Epoch 18/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.1065 - acc: 0.3075 - val_loss: 2.1281 - val_acc: 0.3045 Epoch 19/25 30000/30000 [==============================] - 120s 4ms/step - loss: 2.1000 - acc: 0.3095 - val_loss: 2.1663 - val_acc: 0.3032 Epoch 20/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.0966 - acc: 0.3117 - val_loss: 2.1323 - val_acc: 0.3015 Epoch 21/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.0932 - acc: 0.3124 - val_loss: 2.0983 - val_acc: 0.3143 Epoch 22/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.0936 - acc: 0.3126 - val_loss: 2.1601 - val_acc: 0.3056 Epoch 23/25 30000/30000 [==============================] - 120s 4ms/step - loss: 2.0907 - acc: 0.3134 - val_loss: 2.1049 - val_acc: 0.3065 Epoch 24/25 30000/30000 [==============================] - 119s 4ms/step - loss: 2.0852 - acc: 0.3166 - val_loss: 2.1100 - val_acc: 0.3127 Epoch 25/25 30000/30000 [==============================] - 120s 4ms/step - loss: 2.0868 - acc: 0.3152 - val_loss: 2.1580 - val_acc: 0.2952 Batch size 32: 29.6% (train) vs. 29.5% (validation)
My preconceptions were challenged (yet again!) when I discovered that the low dataset, which I thought could improve the model's performance, could actually be the cause of overfitting. In this case decreasing learnable parameters may have helped to a certain extent, but ultimately the low dataset better (?) enables the model to memorize the training set. Also, as it turns out the softmax function outside the last layer (where I really chose it because it was necessary) suffers from a "vanishing gradient" problem. Because Adam optimizer is an adaptation of the stochastic gradient descent optimization method, this could possibly be why the model failed to learn the large set.
The training of my new model on the large dataset has just concluded. It again appears to have stagnated, even lower than last time. I will stick with this large dataset in spite of my considerations about background noise, because ReLU should be able to account for that. More on this issue down below.
Overfitting has most definitely subsided in its effect. We can see that validation accuracy very closely mirrors training accuracy. So the problem can be reduced to simply fitting the model to the training set.
Let's do a bit of EDA to investigate the "background noise" issue.
plt.imshow(classified_data_2[858][0]) # random index
print(y_data[858])
0.0
plt.imshow(classified_data_2[2839][0]) # random index
print(y_data[2839])
1.0
plt.imshow(classified_data_2[3857][0]) # random index
print(y_data[3857])
1.0
Thoughts:
We can see that there is a strict "dichotomy" between background and object. This was very obvious and intuitive in the case of the small dataset, but it also applies here, albeit in a more subtle manner. A good ReLU activation should be able to capture this dichotomy well enough for us to be able to use the large dataset. Here I think it applies. Training might take a lot longer which is why I should really think through each change I make to the architecture henceforth.
The amount of blur in the background is also quite a lot higher than in the object itself (although of course there is also some blur inside the object as well). The image inside the object appears to be "sharper" than it is on the outside. This is a subjective measure to a human, but we can make the machine learning model "learn" what a blur is. It might just require us to enlarge the kernels of some of the convolutional layers slightly. I do think a ReLU is called for, though. First layer kernel will be enlarged to 5x5 (odd size is still necessary for symmetry purposes)
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=(5, 5, 3), filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=5, input_dim=48, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=5, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2)
tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(classified_data_2[30000:], one_hot_encode(y_data[30000:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 30000 samples, validate on 7500 samples Epoch 1/25 30000/30000 [==============================] - 168s 6ms/step - loss: 2.5727 - acc: 0.1316 - val_loss: 2.4693 - val_acc: 0.1703 Epoch 2/25 30000/30000 [==============================] - 141s 5ms/step - loss: 2.4262 - acc: 0.1898 - val_loss: 2.3720 - val_acc: 0.2151 Epoch 3/25 30000/30000 [==============================] - 142s 5ms/step - loss: 2.3627 - acc: 0.2183 - val_loss: 2.3315 - val_acc: 0.2404 Epoch 4/25 30000/30000 [==============================] - 141s 5ms/step - loss: 2.3134 - acc: 0.2365 - val_loss: 2.2774 - val_acc: 0.2435 Epoch 5/25 30000/30000 [==============================] - 138s 5ms/step - loss: 2.2845 - acc: 0.2428 - val_loss: 2.2926 - val_acc: 0.2372 Epoch 6/25 4064/30000 [===>..........................] - ETA: 1:53 - loss: 2.2552 - acc: 0.2589
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-645-65393ac15e74> in <module>() 19 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 20 ---> 21 second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2) 22 23 tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
Overfitting should not occur because the dataset is quite large (and 27 learnable parameters vs. 75 learnable parameters) within the first convolutional layer is not much of a change especially when considering the 5x48+15x5=315 learnable parameters in the fully connected layers.
I keyboard-interrupted the training process after only five epochs because it didn't seem to be making much progress. If anything its accuracy is slightly lower than when I set the kernel size to 3x3x3 rather than 5x5x3. It might be that it's taking longer to converge because there are more parameters involved.
But I think its low accuracy for all this time is simply because the convolutional architecture is limited and not enough to properly extract features. It is possible to remove the background noise and learn stuff about the object itself; we just need more layers.
Come to think of it, increasing the kernel size shouldn't have much of an effect and just increases the training time pointlessly. Instead I should probably add more convolutional layers that use the ReLU activation. Trustworthy sources from Towards Data Science state that the ReLU allows a neural network to create "folds" around the training set. This sounds like overfitting but done in moderation it (hopefully) shouldn't be a problem.
It could be that the three max-pools are too many and remove crucial parts of the image. I will reduce max-pools to two but put the network through two convolutional layers before each one.
Further, stressing the importance of the dense layers I will increase the number of neurons in the first dense layer to 16 because 5 neurons can cause information to be abstracted away.
Conducting an adequate scientific investigation into the effects of the changes described above will require controlling some of them and changing one thing at a time to understand what's really going on. Let me change the dense layer first.
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=(5, 5, 3), filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=16, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2)
tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(classified_data_2[30000:], one_hot_encode(y_data[30000:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 30000 samples, validate on 7500 samples Epoch 1/25 30000/30000 [==============================] - 170s 6ms/step - loss: 2.4605 - acc: 0.1840 - val_loss: 2.3667 - val_acc: 0.2255 Epoch 2/25 30000/30000 [==============================] - 143s 5ms/step - loss: 2.3052 - acc: 0.2443 - val_loss: 2.2055 - val_acc: 0.2712 Epoch 3/25 30000/30000 [==============================] - 140s 5ms/step - loss: 2.2042 - acc: 0.2754 - val_loss: 2.1720 - val_acc: 0.2827 Epoch 4/25 30000/30000 [==============================] - 142s 5ms/step - loss: 2.1619 - acc: 0.2897 - val_loss: 2.1496 - val_acc: 0.2925 Epoch 5/25 30000/30000 [==============================] - 140s 5ms/step - loss: 2.1372 - acc: 0.3026 - val_loss: 2.1053 - val_acc: 0.3137 Epoch 6/25 30000/30000 [==============================] - 139s 5ms/step - loss: 2.1206 - acc: 0.3051 - val_loss: 2.1162 - val_acc: 0.3112 Epoch 7/25 28320/30000 [===========================>..] - ETA: 7s - loss: 2.1083 - acc: 0.3131
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-646-70b400f29f02> in <module>() 19 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 20 ---> 21 second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2) 22 23 tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
It helps.
I do think that increasing the number of neurons in the first dense layer is a necessary step to take, because five is way too few. 16 seems like a good number.
Let's experiment now with the idea of removing one max-pool and adding additional conv layers.
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=16, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=16, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2)
tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(classified_data_2[30000:], one_hot_encode(y_data[30000:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 30000 samples, validate on 7500 samples Epoch 1/25 30000/30000 [==============================] - 185s 6ms/step - loss: 2.5042 - acc: 0.1660 - val_loss: 2.3361 - val_acc: 0.2489 Epoch 2/25 30000/30000 [==============================] - 151s 5ms/step - loss: 2.2990 - acc: 0.2494 - val_loss: 2.2516 - val_acc: 0.2653 Epoch 3/25 30000/30000 [==============================] - 151s 5ms/step - loss: 2.2004 - acc: 0.2850 - val_loss: 2.1555 - val_acc: 0.3037 Epoch 4/25 30000/30000 [==============================] - 151s 5ms/step - loss: 2.1235 - acc: 0.3100 - val_loss: 2.1079 - val_acc: 0.3191 Epoch 5/25 30000/30000 [==============================] - 150s 5ms/step - loss: 2.0665 - acc: 0.3286 - val_loss: 2.0624 - val_acc: 0.3325 Epoch 6/25 30000/30000 [==============================] - 150s 5ms/step - loss: 2.0316 - acc: 0.3439 - val_loss: 2.0459 - val_acc: 0.3385 Epoch 7/25 30000/30000 [==============================] - 150s 5ms/step - loss: 2.0063 - acc: 0.3494 - val_loss: 2.0284 - val_acc: 0.3436 Epoch 8/25 30000/30000 [==============================] - 151s 5ms/step - loss: 1.9803 - acc: 0.3544 - val_loss: 2.0345 - val_acc: 0.3436 Epoch 9/25 30000/30000 [==============================] - 150s 5ms/step - loss: 1.9637 - acc: 0.3630 - val_loss: 2.0126 - val_acc: 0.3556 Epoch 10/25 30000/30000 [==============================] - 150s 5ms/step - loss: 1.9496 - acc: 0.3654 - val_loss: 1.9834 - val_acc: 0.3643 Epoch 11/25 30000/30000 [==============================] - 150s 5ms/step - loss: 1.9334 - acc: 0.3742 - val_loss: 1.9790 - val_acc: 0.3589 Epoch 12/25 30000/30000 [==============================] - 151s 5ms/step - loss: 1.9237 - acc: 0.3758 - val_loss: 1.9867 - val_acc: 0.3583 Epoch 13/25 3136/30000 [==>...........................] - ETA: 2:21 - loss: 1.9266 - acc: 0.3734
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-647-25322cd10127> in <module>() 19 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 20 ---> 21 second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2) 22 23 tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
It helps more.
I have the idea of adding more neurons to the dense layer. More filters for convolutional layers in order to better the feature extraction.
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=5, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2)
tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(classified_data_2[30000:], one_hot_encode(y_data[30000:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 30000 samples, validate on 7500 samples Epoch 1/25 30000/30000 [==============================] - 192s 6ms/step - loss: 2.3333 - acc: 0.2359 - val_loss: 2.1282 - val_acc: 0.3112 Epoch 2/25 30000/30000 [==============================] - 161s 5ms/step - loss: 2.0988 - acc: 0.3178 - val_loss: 2.0704 - val_acc: 0.3301 Epoch 3/25 30000/30000 [==============================] - 163s 5ms/step - loss: 1.9880 - acc: 0.3525 - val_loss: 2.0456 - val_acc: 0.3472 Epoch 4/25 30000/30000 [==============================] - 161s 5ms/step - loss: 1.9236 - acc: 0.3747 - val_loss: 1.9531 - val_acc: 0.3732 Epoch 5/25 30000/30000 [==============================] - 169s 6ms/step - loss: 1.8703 - acc: 0.3928 - val_loss: 1.9469 - val_acc: 0.3687 Epoch 6/25 416/30000 [..............................] - ETA: 2:26 - loss: 1.9036 - acc: 0.3966
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-650-4382711f3db4> in <module>() 19 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 20 ---> 21 second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2) 22 23 tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
The results from adding more filters are promising indicating that it was a problem with feature extraction. This makes a lot of sense because there were already many neurons in the dense layers (it's not shown, but I trained another model with only 5 filters but 64 neurons in the first fully-connected layer, and it got a considerably worse performance. My logic was that more filters are needed to capture lower-level features. You don't need as many for higher-level, although perhaps I should add more. In fact, let's try some of that.
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2)
tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(classified_data_2[30000:], one_hot_encode(y_data[30000:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 30000 samples, validate on 7500 samples Epoch 1/25 30000/30000 [==============================] - 303s 10ms/step - loss: 2.3212 - acc: 0.2453 - val_loss: 2.0181 - val_acc: 0.3539 Epoch 2/25 30000/30000 [==============================] - 276s 9ms/step - loss: 1.9750 - acc: 0.3598 - val_loss: 1.9387 - val_acc: 0.3833 Epoch 3/25 30000/30000 [==============================] - 274s 9ms/step - loss: 1.8275 - acc: 0.4100 - val_loss: 1.8480 - val_acc: 0.4196 Epoch 4/25 30000/30000 [==============================] - 272s 9ms/step - loss: 1.7171 - acc: 0.4467 - val_loss: 1.8388 - val_acc: 0.4136 Epoch 5/25 30000/30000 [==============================] - 281s 9ms/step - loss: 1.6287 - acc: 0.4734 - val_loss: 1.7959 - val_acc: 0.4365 Epoch 6/25 30000/30000 [==============================] - 271s 9ms/step - loss: 1.5412 - acc: 0.4990 - val_loss: 1.8177 - val_acc: 0.4368 Epoch 7/25 30000/30000 [==============================] - 269s 9ms/step - loss: 1.4648 - acc: 0.5230 - val_loss: 1.7819 - val_acc: 0.4449 Epoch 8/25 30000/30000 [==============================] - 270s 9ms/step - loss: 1.3976 - acc: 0.5382 - val_loss: 1.8589 - val_acc: 0.4283 Epoch 9/25 30000/30000 [==============================] - 267s 9ms/step - loss: 1.3304 - acc: 0.5621 - val_loss: 1.8850 - val_acc: 0.4289 Epoch 10/25 1120/30000 [>.............................] - ETA: 4:22 - loss: 1.2132 - acc: 0.6054
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-652-b94bdd830696> in <module>() 19 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 20 ---> 21 second_iteration.fit(classified_data_2, one_hot_encode(y_data), batch_size=32, epochs=25, validation_split=0.2) 22 23 tr_acc = second_iteration.evaluate(classified_data_2[:30000], one_hot_encode(y_data[:30000]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
The results from adding more filters to the convolutional layers are very promising indeed! We broke the 40% barrier for the very first time on the validation set. I don't think this change should cause significant overfitting because 1) the dataset is large enough that feasibly training a model to fit to it within a low number of epochs is not doable, and 2) the kernels are only 3x3x3. Adding 22 more kernels (in the case of the first layer) barely adds any learnable parameters compared to the thousands of parameters added to the dense layers. Things are looking up!
UPD: Some small amounts of overfitting were detected. I suppose that reducing the number of kernels in convolutional layers to 24/12 instead of 32/16 could partially mitigate this issue, as it was not present with a 10/5 split.
Tomorrow there will be a meeting with an AI expert (tomorrow being 23 November). Things to discuss:
Insist on the fact that this project is a mechanism to learn about neural networks in general.
UPD2: Clearly there is significant overfitting happening which has to be directly caused by the additional kernels.
New best validation accuracy: 44.5%
Interview:
Works at the International Computing Center for the United Nations. Name: Adrian Errea
Notes:
Elaboration on "error analysis". Choose some samples and run the model, see what type of sample it struggles with the most and factor in to decision-making
Training on smaller datasets should not cause too much overfitting, it allows for more epochs and deeper networks
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=50, validation_split=0.2)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 660 samples, validate on 165 samples Epoch 1/50 660/660 [==============================] - 52s 78ms/step - loss: 2.8140 - acc: 0.1000 - val_loss: 2.6514 - val_acc: 0.1273 Epoch 2/50 660/660 [==============================] - 9s 14ms/step - loss: 2.5721 - acc: 0.1621 - val_loss: 2.5170 - val_acc: 0.2121 Epoch 3/50 660/660 [==============================] - 8s 12ms/step - loss: 2.3672 - acc: 0.2530 - val_loss: 2.4295 - val_acc: 0.2424 Epoch 4/50 660/660 [==============================] - 8s 12ms/step - loss: 2.0778 - acc: 0.3242 - val_loss: 2.4615 - val_acc: 0.2667 Epoch 5/50 660/660 [==============================] - 7s 11ms/step - loss: 1.8529 - acc: 0.4076 - val_loss: 2.3376 - val_acc: 0.3091 Epoch 6/50 660/660 [==============================] - 7s 11ms/step - loss: 1.6008 - acc: 0.4803 - val_loss: 2.4765 - val_acc: 0.3212 Epoch 7/50 660/660 [==============================] - 7s 11ms/step - loss: 1.3518 - acc: 0.5848 - val_loss: 2.6166 - val_acc: 0.3030 Epoch 8/50 660/660 [==============================] - 8s 11ms/step - loss: 0.9984 - acc: 0.6833 - val_loss: 3.0411 - val_acc: 0.3030 Epoch 9/50 660/660 [==============================] - 8s 11ms/step - loss: 0.7636 - acc: 0.7530 - val_loss: 3.0313 - val_acc: 0.2606 Epoch 10/50 660/660 [==============================] - 7s 11ms/step - loss: 0.5313 - acc: 0.8424 - val_loss: 3.7024 - val_acc: 0.3152 Epoch 11/50 660/660 [==============================] - 7s 10ms/step - loss: 0.3685 - acc: 0.8727 - val_loss: 4.2417 - val_acc: 0.2727 Epoch 12/50 660/660 [==============================] - 7s 11ms/step - loss: 0.2958 - acc: 0.9000 - val_loss: 3.9853 - val_acc: 0.3152 Epoch 13/50 660/660 [==============================] - 7s 11ms/step - loss: 0.1843 - acc: 0.9515 - val_loss: 4.4062 - val_acc: 0.3455 Epoch 14/50 660/660 [==============================] - 6s 10ms/step - loss: 0.2050 - acc: 0.9379 - val_loss: 4.6885 - val_acc: 0.2727 Epoch 15/50 660/660 [==============================] - 7s 10ms/step - loss: 0.0951 - acc: 0.9742 - val_loss: 5.1160 - val_acc: 0.3152 Epoch 16/50 660/660 [==============================] - 8s 12ms/step - loss: 0.0501 - acc: 0.9833 - val_loss: 5.4748 - val_acc: 0.3030 Epoch 17/50 660/660 [==============================] - 8s 12ms/step - loss: 0.0767 - acc: 0.9818 - val_loss: 5.4729 - val_acc: 0.3152 Epoch 18/50 660/660 [==============================] - 8s 11ms/step - loss: 0.0467 - acc: 0.9864 - val_loss: 5.6996 - val_acc: 0.3515 Epoch 19/50 660/660 [==============================] - 7s 11ms/step - loss: 0.0166 - acc: 1.0000 - val_loss: 5.8649 - val_acc: 0.3515 Epoch 20/50 660/660 [==============================] - 8s 12ms/step - loss: 0.0066 - acc: 1.0000 - val_loss: 6.0445 - val_acc: 0.3455 Epoch 21/50 660/660 [==============================] - 7s 11ms/step - loss: 0.0071 - acc: 0.9985 - val_loss: 6.2744 - val_acc: 0.3333 Epoch 22/50 660/660 [==============================] - 7s 11ms/step - loss: 0.0032 - acc: 1.0000 - val_loss: 6.3784 - val_acc: 0.3273 Epoch 23/50 660/660 [==============================] - 7s 11ms/step - loss: 0.0018 - acc: 1.0000 - val_loss: 6.4475 - val_acc: 0.3515 Epoch 24/50 660/660 [==============================] - 7s 10ms/step - loss: 0.0014 - acc: 1.0000 - val_loss: 6.5400 - val_acc: 0.3515 Epoch 25/50 660/660 [==============================] - 7s 10ms/step - loss: 0.0011 - acc: 1.0000 - val_loss: 6.6292 - val_acc: 0.3455 Epoch 26/50 660/660 [==============================] - 7s 11ms/step - loss: 8.8368e-04 - acc: 1.0000 - val_loss: 6.6901 - val_acc: 0.3455 Epoch 27/50 660/660 [==============================] - 7s 10ms/step - loss: 7.5889e-04 - acc: 1.0000 - val_loss: 6.7199 - val_acc: 0.3455 Epoch 28/50 660/660 [==============================] - 6s 10ms/step - loss: 6.9580e-04 - acc: 1.0000 - val_loss: 6.8049 - val_acc: 0.3455 Epoch 29/50 660/660 [==============================] - 6s 10ms/step - loss: 5.9060e-04 - acc: 1.0000 - val_loss: 6.8423 - val_acc: 0.3394 Epoch 30/50 660/660 [==============================] - 6s 10ms/step - loss: 5.2662e-04 - acc: 1.0000 - val_loss: 6.8833 - val_acc: 0.3455 Epoch 31/50 660/660 [==============================] - 7s 10ms/step - loss: 4.7614e-04 - acc: 1.0000 - val_loss: 6.9167 - val_acc: 0.3394 Epoch 32/50 660/660 [==============================] - 7s 10ms/step - loss: 4.3864e-04 - acc: 1.0000 - val_loss: 6.9443 - val_acc: 0.3455 Epoch 33/50 660/660 [==============================] - 7s 10ms/step - loss: 4.0363e-04 - acc: 1.0000 - val_loss: 7.0101 - val_acc: 0.3515 Epoch 34/50 660/660 [==============================] - 6s 9ms/step - loss: 3.6813e-04 - acc: 1.0000 - val_loss: 7.0328 - val_acc: 0.3455 Epoch 35/50 660/660 [==============================] - 6s 9ms/step - loss: 3.3913e-04 - acc: 1.0000 - val_loss: 7.0487 - val_acc: 0.3455 Epoch 36/50 660/660 [==============================] - 7s 11ms/step - loss: 3.1809e-04 - acc: 1.0000 - val_loss: 7.0663 - val_acc: 0.3455 Epoch 37/50 660/660 [==============================] - 7s 11ms/step - loss: 3.0197e-04 - acc: 1.0000 - val_loss: 7.1057 - val_acc: 0.3455 Epoch 38/50 660/660 [==============================] - 7s 11ms/step - loss: 2.7734e-04 - acc: 1.0000 - val_loss: 7.1223 - val_acc: 0.3455 Epoch 39/50 660/660 [==============================] - 7s 10ms/step - loss: 2.6144e-04 - acc: 1.0000 - val_loss: 7.1412 - val_acc: 0.3455 Epoch 40/50 660/660 [==============================] - 7s 10ms/step - loss: 2.4795e-04 - acc: 1.0000 - val_loss: 7.1641 - val_acc: 0.3455 Epoch 41/50 660/660 [==============================] - 7s 10ms/step - loss: 2.3684e-04 - acc: 1.0000 - val_loss: 7.1686 - val_acc: 0.3455 Epoch 42/50 660/660 [==============================] - 7s 11ms/step - loss: 2.1844e-04 - acc: 1.0000 - val_loss: 7.1873 - val_acc: 0.3455 Epoch 43/50 660/660 [==============================] - 7s 11ms/step - loss: 2.0615e-04 - acc: 1.0000 - val_loss: 7.2276 - val_acc: 0.3455 Epoch 44/50 660/660 [==============================] - 7s 11ms/step - loss: 1.9800e-04 - acc: 1.0000 - val_loss: 7.2403 - val_acc: 0.3455 Epoch 45/50 660/660 [==============================] - 7s 11ms/step - loss: 1.8667e-04 - acc: 1.0000 - val_loss: 7.2494 - val_acc: 0.3455 Epoch 46/50 660/660 [==============================] - 7s 11ms/step - loss: 1.7791e-04 - acc: 1.0000 - val_loss: 7.2683 - val_acc: 0.3455 Epoch 47/50 660/660 [==============================] - 7s 11ms/step - loss: 1.6772e-04 - acc: 1.0000 - val_loss: 7.2796 - val_acc: 0.3455 Epoch 48/50 660/660 [==============================] - 7s 11ms/step - loss: 1.6142e-04 - acc: 1.0000 - val_loss: 7.2875 - val_acc: 0.3455 Epoch 49/50 660/660 [==============================] - 8s 11ms/step - loss: 1.6013e-04 - acc: 1.0000 - val_loss: 7.3130 - val_acc: 0.3455 Epoch 50/50 660/660 [==============================] - 7s 11ms/step - loss: 1.4660e-04 - acc: 1.0000 - val_loss: 7.3225 - val_acc: 0.3455 Batch size 32: 99.9% (train) vs. 34.5% (validation)
#from tensorflow.keras import initializers as init
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=15, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=50, validation_split=0.2)
tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(transposed_white_data[660:], one_hot_encode(white_y[660:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 660 samples, validate on 165 samples Epoch 1/50 660/660 [==============================] - 39s 60ms/step - loss: 2.7153 - acc: 0.0682 - val_loss: 2.6875 - val_acc: 0.0667 Epoch 2/50 660/660 [==============================] - 4s 6ms/step - loss: 2.6128 - acc: 0.1485 - val_loss: 2.4522 - val_acc: 0.2485 Epoch 3/50 660/660 [==============================] - 4s 6ms/step - loss: 2.3669 - acc: 0.2318 - val_loss: 2.3814 - val_acc: 0.2848 Epoch 4/50 660/660 [==============================] - 4s 6ms/step - loss: 2.1538 - acc: 0.3121 - val_loss: 2.2710 - val_acc: 0.2970 Epoch 5/50 660/660 [==============================] - 4s 6ms/step - loss: 1.8881 - acc: 0.4076 - val_loss: 2.3102 - val_acc: 0.2606 Epoch 6/50 660/660 [==============================] - 4s 6ms/step - loss: 1.6547 - acc: 0.4773 - val_loss: 2.3601 - val_acc: 0.2970 Epoch 7/50 660/660 [==============================] - 4s 6ms/step - loss: 1.3873 - acc: 0.5545 - val_loss: 2.5893 - val_acc: 0.3333 Epoch 8/50 660/660 [==============================] - 4s 6ms/step - loss: 1.1361 - acc: 0.6561 - val_loss: 2.9017 - val_acc: 0.3030 Epoch 9/50 660/660 [==============================] - 4s 6ms/step - loss: 0.8717 - acc: 0.7197 - val_loss: 2.9037 - val_acc: 0.3091 Epoch 10/50 660/660 [==============================] - 4s 6ms/step - loss: 0.6457 - acc: 0.7985 - val_loss: 3.1907 - val_acc: 0.3515 Epoch 11/50 660/660 [==============================] - 4s 6ms/step - loss: 0.4962 - acc: 0.8470 - val_loss: 3.6492 - val_acc: 0.2970 Epoch 12/50 660/660 [==============================] - 4s 6ms/step - loss: 0.3219 - acc: 0.9091 - val_loss: 4.2007 - val_acc: 0.3030 Epoch 13/50 660/660 [==============================] - 4s 6ms/step - loss: 0.2111 - acc: 0.9394 - val_loss: 4.8649 - val_acc: 0.3152 Epoch 14/50 660/660 [==============================] - 4s 6ms/step - loss: 0.1769 - acc: 0.9530 - val_loss: 5.2783 - val_acc: 0.2909 Epoch 15/50 660/660 [==============================] - 4s 6ms/step - loss: 0.1222 - acc: 0.9682 - val_loss: 5.5983 - val_acc: 0.3455 Epoch 16/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0711 - acc: 0.9818 - val_loss: 5.8715 - val_acc: 0.3030 Epoch 17/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0440 - acc: 0.9924 - val_loss: 6.3212 - val_acc: 0.2545 Epoch 18/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0265 - acc: 0.9955 - val_loss: 6.6976 - val_acc: 0.2667 Epoch 19/50 660/660 [==============================] - 4s 6ms/step - loss: 0.1850 - acc: 0.9424 - val_loss: 5.5078 - val_acc: 0.3152 Epoch 20/50 660/660 [==============================] - 4s 6ms/step - loss: 0.1645 - acc: 0.9500 - val_loss: 5.7478 - val_acc: 0.2970 Epoch 21/50 660/660 [==============================] - 4s 6ms/step - loss: 0.1191 - acc: 0.9667 - val_loss: 5.8233 - val_acc: 0.3273 Epoch 22/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0745 - acc: 0.9803 - val_loss: 5.9015 - val_acc: 0.2606 Epoch 23/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0254 - acc: 0.9939 - val_loss: 6.2201 - val_acc: 0.2909 Epoch 24/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0119 - acc: 0.9985 - val_loss: 6.4386 - val_acc: 0.3030 Epoch 25/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0057 - acc: 1.0000 - val_loss: 6.5537 - val_acc: 0.3152 Epoch 26/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0033 - acc: 1.0000 - val_loss: 6.5881 - val_acc: 0.3152 Epoch 27/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0023 - acc: 1.0000 - val_loss: 6.6881 - val_acc: 0.3152 Epoch 28/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0017 - acc: 1.0000 - val_loss: 6.7744 - val_acc: 0.3212 Epoch 29/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0016 - acc: 1.0000 - val_loss: 6.8492 - val_acc: 0.3152 Epoch 30/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0013 - acc: 1.0000 - val_loss: 6.8960 - val_acc: 0.3152 Epoch 31/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0011 - acc: 1.0000 - val_loss: 6.9730 - val_acc: 0.3152 Epoch 32/50 660/660 [==============================] - 4s 6ms/step - loss: 0.0010 - acc: 1.0000 - val_loss: 7.0223 - val_acc: 0.3212 Epoch 33/50 660/660 [==============================] - 4s 6ms/step - loss: 8.5759e-04 - acc: 1.0000 - val_loss: 7.0521 - val_acc: 0.3091 Epoch 34/50 320/660 [=============>................] - ETA: 1s - loss: 4.5169e-04 - acc: 1.0000
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-656-9803fdc7bb58> in <module>() 19 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 20 ---> 21 second_iteration.fit(transposed_white_data, one_hot_encode(white_y), batch_size=32, epochs=50, validation_split=0.2) 22 23 tr_acc = second_iteration.evaluate(transposed_white_data[:660], one_hot_encode(white_y[:660]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
The CNN that I am training has memorized the entire training set by heart, which I find very funny but is also bad news for our project. Lowering the number of filters doesn't help.
The fact that the CNN learnt the entire set by heart is not necessarily indicative of its ability to learn features that it can sustain over time, but rather it is just memorizing data. This is extremely severe overfitting and it needs to be fixed.
What samples does it struggle with?
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', 'carnivores', \
'omni/herbivores', 'small/medium mammals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 164)
plt.imshow(transposed_white_data[660 + i][0])
l = second_iteration.predict(transposed_white_data[660 + i:661 + i])
ind = [j for j in range(15) if l[0][j] == max(l[0])]
second = [j for j in range(15) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[white_y[660 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
Expected: people Got: electronics Confidence: 0.9976966977119446 Second option: furniture Second confidence: 0.001933321007527411
I ran the above experiment about 20-30 times (granted, it was quite fun) and I made some very useful observations about the model.
Plan: merge "carnivores", "omni/herbivores", and "small to medium mammals" into one class (called "land animals"), which means total 13 classes. Now take 300 random samples from each class to form train + validation sets. 80-20 split means 3120 training samples and 780 validation samples.
condensed_data = np.zeros((3900, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((3900), dtype=np.uint8)
ptrs = [300] * 13
training = 0
validation = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [8, 9, 10]: val = 8
elif val in [11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
if ptrs[val] <= 60:
condensed_data[3120 + validation] = classified_data_2[i]
condensed_y[3120 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
#from tensorflow.keras import initializers as init
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=13, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(condensed_data, one_hot_encode(condensed_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=True)
tr_acc = second_iteration.evaluate(condensed_data[:3120], one_hot_encode(condensed_y[:3120]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(condensed_data[3120:], one_hot_encode(condensed_y[3120:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 3120 samples, validate on 780 samples Epoch 1/50 3120/3120 [==============================] - 56s 18ms/step - loss: 2.5123 - acc: 0.1093 - val_loss: 2.4156 - val_acc: 0.1821 Epoch 2/50 3120/3120 [==============================] - 21s 7ms/step - loss: 2.3812 - acc: 0.1817 - val_loss: 2.2835 - val_acc: 0.1974 Epoch 3/50 3120/3120 [==============================] - 19s 6ms/step - loss: 2.2645 - acc: 0.2160 - val_loss: 2.2251 - val_acc: 0.2385 Epoch 4/50 3120/3120 [==============================] - 18s 6ms/step - loss: 2.1665 - acc: 0.2651 - val_loss: 2.2377 - val_acc: 0.2321 Epoch 5/50 3120/3120 [==============================] - 18s 6ms/step - loss: 2.1231 - acc: 0.2830 - val_loss: 2.2273 - val_acc: 0.2487 Epoch 6/50 3120/3120 [==============================] - 18s 6ms/step - loss: 2.0264 - acc: 0.3212 - val_loss: 2.1045 - val_acc: 0.3090 Epoch 7/50 3120/3120 [==============================] - 18s 6ms/step - loss: 1.9513 - acc: 0.3545 - val_loss: 2.1281 - val_acc: 0.2718 Epoch 8/50 3120/3120 [==============================] - 18s 6ms/step - loss: 1.8552 - acc: 0.3766 - val_loss: 2.1486 - val_acc: 0.3000 Epoch 9/50 3120/3120 [==============================] - 18s 6ms/step - loss: 1.7668 - acc: 0.4074 - val_loss: 2.2266 - val_acc: 0.3128 Epoch 10/50 768/3120 [======>.......................] - ETA: 13s - loss: 1.6483 - acc: 0.4401
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1075-55ac2c322ef0> in <module>() 25 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 26 ---> 27 second_iteration.fit(condensed_data, one_hot_encode(condensed_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=True) 28 29 tr_acc = second_iteration.evaluate(condensed_data[:3120], one_hot_encode(condensed_y[:3120]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 779)
plt.imshow(condensed_data[3120 + i][0])
l = second_iteration.predict(condensed_data[3120 + i:3121 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[condensed_y[3120 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
Expected: trees Got: trees Confidence: 0.7485129833221436 Second option: land animals Second confidence: 0.0995328277349472
Overfitting still occurs. But at least now we are using a dataset with more general images and the three similar classes of "carnivores", "omni/herbivores" and "small to medium sized mammals" are now condensed into one class.
Let's try to solve the overfitting problem using dropout. This way, we ensure that the model fully learns features rather than noise.
Source: https://towardsdatascience.com/dropout-in-neural-networks-47a162d621d9, https://machinelearningmastery.com/dropout-regularization-deep-learning-models-keras/
#from tensorflow.keras import initializers as init
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(condensed_data, one_hot_encode(condensed_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=True)
tr_acc = second_iteration.evaluate(condensed_data[:3120], one_hot_encode(condensed_y[:3120]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(condensed_data[3120:], one_hot_encode(condensed_y[3120:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 3120 samples, validate on 780 samples Epoch 1/50 3120/3120 [==============================] - 55s 18ms/step - loss: 2.5257 - acc: 0.1067 - val_loss: 2.4337 - val_acc: 0.1667 Epoch 2/50 3120/3120 [==============================] - 19s 6ms/step - loss: 2.3557 - acc: 0.1984 - val_loss: 2.2969 - val_acc: 0.2321 Epoch 3/50 3120/3120 [==============================] - 19s 6ms/step - loss: 2.1923 - acc: 0.2654 - val_loss: 2.1385 - val_acc: 0.2846 Epoch 4/50 3120/3120 [==============================] - 19s 6ms/step - loss: 2.0608 - acc: 0.3199 - val_loss: 2.1477 - val_acc: 0.2936 Epoch 5/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.9868 - acc: 0.3391 - val_loss: 2.1586 - val_acc: 0.2808 Epoch 6/50 512/3120 [===>..........................] - ETA: 14s - loss: 1.9309 - acc: 0.3730
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1108-0f27c3512f6a> in <module>() 27 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 28 ---> 29 second_iteration.fit(condensed_data, one_hot_encode(condensed_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=True) 30 31 tr_acc = second_iteration.evaluate(condensed_data[:3120], one_hot_encode(condensed_y[:3120]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
It seems that no matter what I change, at some point the loss function in validation starts getting reverse-optimized. Or more formally, just increasing after some period of decrease.
This is an indicator of overfitting, but it's just as much an indicator that it's probably an underlying problem with the data that is causing this issue. I'll run the investigation again.
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 779)
plt.imshow(condensed_data[3120 + i][0])
l = second_iteration.predict(condensed_data[3120 + i:3121 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[condensed_y[3120 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
Expected: food containers Got: land animals Confidence: 0.19979310035705566 Second option: food containers Second confidence: 0.17681020498275757
#from tensorflow.keras import initializers as init
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(condensed_data, one_hot_encode(condensed_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=True)
tr_acc = second_iteration.evaluate(condensed_data[:3120], one_hot_encode(condensed_y[:3120]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(condensed_data[3120:], one_hot_encode(condensed_y[3120:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 3120 samples, validate on 780 samples Epoch 1/50 3120/3120 [==============================] - 76s 24ms/step - loss: 2.5330 - acc: 0.0965 - val_loss: 2.4717 - val_acc: 0.1500 Epoch 2/50 3120/3120 [==============================] - 19s 6ms/step - loss: 2.3987 - acc: 0.1862 - val_loss: 2.2782 - val_acc: 0.2167 Epoch 3/50 3120/3120 [==============================] - 19s 6ms/step - loss: 2.2334 - acc: 0.2490 - val_loss: 2.1603 - val_acc: 0.2718 Epoch 4/50 3120/3120 [==============================] - 19s 6ms/step - loss: 2.1307 - acc: 0.2849 - val_loss: 2.1376 - val_acc: 0.2859 Epoch 5/50 3120/3120 [==============================] - 21s 7ms/step - loss: 2.0270 - acc: 0.3234 - val_loss: 2.0408 - val_acc: 0.3359 Epoch 6/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.9240 - acc: 0.3538 - val_loss: 1.9810 - val_acc: 0.3474 Epoch 7/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.8539 - acc: 0.3654 - val_loss: 2.0411 - val_acc: 0.3256 Epoch 8/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.7938 - acc: 0.3926 - val_loss: 2.0177 - val_acc: 0.3141 Epoch 9/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.7163 - acc: 0.4215 - val_loss: 1.9691 - val_acc: 0.3538 Epoch 10/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.6352 - acc: 0.4490 - val_loss: 1.9954 - val_acc: 0.3513 Epoch 11/50 416/3120 [===>..........................] - ETA: 15s - loss: 1.5986 - acc: 0.4663
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1149-9db5c35e127a> in <module>() 27 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 28 ---> 29 second_iteration.fit(condensed_data, one_hot_encode(condensed_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=True) 30 31 tr_acc = second_iteration.evaluate(condensed_data[:3120], one_hot_encode(condensed_y[:3120]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
Today is Friday, 24th of November. The goal is to reduce overfitting. Previous attempts to use dropout didn't work too well, so let's try different approaches.
Firstly I'm not even sure whether "shuffle" will shuffle the entire set, or split 80-20 and then shuffle. To be completely sure...
shuffled_data = np.zeros((3900, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((3900), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(3120)]
shuffled_valid = [*range(3120, 3900)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(3120):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(780):
shuffled_data[shuffled_valid[i]] = condensed_data[3120 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[3120 + i]
#from tensorflow.keras import initializers as init
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[3120:], one_hot_encode(shuffled_y[3120:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 3120 samples, validate on 780 samples Epoch 1/50 3120/3120 [==============================] - 65s 21ms/step - loss: 2.4699 - acc: 0.1404 - val_loss: 2.3212 - val_acc: 0.2167 Epoch 2/50 3120/3120 [==============================] - 24s 8ms/step - loss: 2.3142 - acc: 0.2160 - val_loss: 2.2889 - val_acc: 0.2218 Epoch 3/50 3120/3120 [==============================] - 24s 8ms/step - loss: 2.2393 - acc: 0.2404 - val_loss: 2.1788 - val_acc: 0.2744 Epoch 4/50 3120/3120 [==============================] - 24s 8ms/step - loss: 2.1672 - acc: 0.2628 - val_loss: 2.1835 - val_acc: 0.2603 Epoch 5/50 3120/3120 [==============================] - 20s 6ms/step - loss: 2.0970 - acc: 0.2929 - val_loss: 2.1267 - val_acc: 0.2782 Epoch 6/50 3120/3120 [==============================] - 20s 6ms/step - loss: 2.0313 - acc: 0.3192 - val_loss: 2.1412 - val_acc: 0.2769 Epoch 7/50 3120/3120 [==============================] - 20s 7ms/step - loss: 2.0054 - acc: 0.3272 - val_loss: 2.0956 - val_acc: 0.2821 Epoch 8/50 3120/3120 [==============================] - 21s 7ms/step - loss: 1.9594 - acc: 0.3529 - val_loss: 2.0712 - val_acc: 0.2974 Epoch 9/50 3120/3120 [==============================] - 23s 7ms/step - loss: 1.8713 - acc: 0.3696 - val_loss: 2.0886 - val_acc: 0.3154 Epoch 10/50 3120/3120 [==============================] - 22s 7ms/step - loss: 1.8409 - acc: 0.3885 - val_loss: 2.0596 - val_acc: 0.3064 Epoch 11/50 3120/3120 [==============================] - 22s 7ms/step - loss: 1.7930 - acc: 0.3808 - val_loss: 2.0982 - val_acc: 0.2987 Epoch 12/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.7227 - acc: 0.4093 - val_loss: 2.0799 - val_acc: 0.3205 Epoch 13/50 3120/3120 [==============================] - 21s 7ms/step - loss: 1.7036 - acc: 0.4337 - val_loss: 2.1050 - val_acc: 0.2974 Epoch 14/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.6719 - acc: 0.4388 - val_loss: 2.0884 - val_acc: 0.3090 Epoch 15/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.6253 - acc: 0.4449 - val_loss: 2.1772 - val_acc: 0.3051 Epoch 16/50 608/3120 [====>.........................] - ETA: 15s - loss: 1.6179 - acc: 0.4490
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1155-e832d003677b> in <module>() 27 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 28 ---> 29 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 30 31 tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
#from tensorflow.keras import initializers as init
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=64, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=64, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=64, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[3120:], one_hot_encode(shuffled_y[3120:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 3120 samples, validate on 780 samples Epoch 1/50 3120/3120 [==============================] - 61s 20ms/step - loss: 2.5565 - acc: 0.1010 - val_loss: 2.4730 - val_acc: 0.1692 Epoch 2/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.4080 - acc: 0.1744 - val_loss: 2.3245 - val_acc: 0.2064 Epoch 3/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.3069 - acc: 0.2221 - val_loss: 2.2039 - val_acc: 0.2590 Epoch 4/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.2053 - acc: 0.2548 - val_loss: 2.1877 - val_acc: 0.2808 Epoch 5/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.1339 - acc: 0.2792 - val_loss: 2.1346 - val_acc: 0.3026 Epoch 6/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.0733 - acc: 0.3083 - val_loss: 2.1131 - val_acc: 0.3115 Epoch 7/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.0006 - acc: 0.3327 - val_loss: 2.0590 - val_acc: 0.3192 Epoch 8/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.9330 - acc: 0.3561 - val_loss: 2.0506 - val_acc: 0.3244 Epoch 9/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.8671 - acc: 0.3644 - val_loss: 2.0544 - val_acc: 0.3167 Epoch 10/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.8615 - acc: 0.3728 - val_loss: 2.0795 - val_acc: 0.3051 Epoch 11/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.7943 - acc: 0.4010 - val_loss: 1.9874 - val_acc: 0.3538 Epoch 12/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.7137 - acc: 0.4266 - val_loss: 1.9994 - val_acc: 0.3423 Epoch 13/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.6992 - acc: 0.4397 - val_loss: 2.0786 - val_acc: 0.3205 Epoch 14/50 3120/3120 [==============================] - 21s 7ms/step - loss: 1.6311 - acc: 0.4522 - val_loss: 1.9893 - val_acc: 0.3628 Epoch 15/50 3120/3120 [==============================] - 21s 7ms/step - loss: 1.5957 - acc: 0.4612 - val_loss: 2.0122 - val_acc: 0.3423 Epoch 16/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.5519 - acc: 0.4814 - val_loss: 2.1104 - val_acc: 0.3346 Epoch 17/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.4943 - acc: 0.4946 - val_loss: 2.0012 - val_acc: 0.3641 Epoch 18/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.4724 - acc: 0.4955 - val_loss: 2.1123 - val_acc: 0.3372 Epoch 19/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.4659 - acc: 0.4939 - val_loss: 2.0213 - val_acc: 0.3731 Epoch 20/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.4527 - acc: 0.5208 - val_loss: 2.0641 - val_acc: 0.3821 Epoch 21/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.4190 - acc: 0.5292 - val_loss: 2.0708 - val_acc: 0.3679 Epoch 22/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.3727 - acc: 0.5494 - val_loss: 2.1168 - val_acc: 0.3679 Epoch 23/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.3562 - acc: 0.5282 - val_loss: 2.1111 - val_acc: 0.3654 Epoch 24/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.3051 - acc: 0.5471 - val_loss: 2.1628 - val_acc: 0.3628 Epoch 25/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.2965 - acc: 0.5628 - val_loss: 2.0823 - val_acc: 0.3756 Epoch 26/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.2645 - acc: 0.5731 - val_loss: 2.1315 - val_acc: 0.3474 Epoch 27/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.2173 - acc: 0.5801 - val_loss: 2.1198 - val_acc: 0.3538 Epoch 28/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.1575 - acc: 0.6016 - val_loss: 2.2435 - val_acc: 0.3487 Epoch 29/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.1795 - acc: 0.6006 - val_loss: 2.2459 - val_acc: 0.3385 Epoch 30/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.1705 - acc: 0.6016 - val_loss: 2.2656 - val_acc: 0.3487 Epoch 31/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.1052 - acc: 0.6279 - val_loss: 2.3644 - val_acc: 0.3436 Epoch 32/50 3120/3120 [==============================] - 17s 6ms/step - loss: 1.1217 - acc: 0.6167 - val_loss: 2.3653 - val_acc: 0.3423 Epoch 33/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.0733 - acc: 0.6337 - val_loss: 2.3425 - val_acc: 0.3526 Epoch 34/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.0041 - acc: 0.6554 - val_loss: 2.4072 - val_acc: 0.3577 Epoch 35/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.0164 - acc: 0.6417 - val_loss: 2.4538 - val_acc: 0.3410 Epoch 36/50 2240/3120 [====================>.........] - ETA: 4s - loss: 0.9971 - acc: 0.6612
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1156-8d7c765b6109> in <module>() 27 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 28 ---> 29 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=64, epochs=50, validation_split=0.2, shuffle=False) 30 31 tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
We can see that the overfitting is somewhat mitigated when we increase the batch size. But it still manages 65% accuracy on training set.
#from tensorflow.keras import initializers as init
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=64, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[3120:], one_hot_encode(shuffled_y[3120:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 3120 samples, validate on 780 samples Epoch 1/50 3120/3120 [==============================] - 61s 19ms/step - loss: 2.5438 - acc: 0.0971 - val_loss: 2.4461 - val_acc: 0.1474 Epoch 2/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.4103 - acc: 0.1715 - val_loss: 2.3169 - val_acc: 0.1936 Epoch 3/50 3120/3120 [==============================] - 15s 5ms/step - loss: 2.3280 - acc: 0.1923 - val_loss: 2.2625 - val_acc: 0.2256 Epoch 4/50 3120/3120 [==============================] - 15s 5ms/step - loss: 2.2669 - acc: 0.2103 - val_loss: 2.1950 - val_acc: 0.2500 Epoch 5/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.2003 - acc: 0.2423 - val_loss: 2.1321 - val_acc: 0.2756 Epoch 6/50 3120/3120 [==============================] - 15s 5ms/step - loss: 2.1406 - acc: 0.2737 - val_loss: 2.1172 - val_acc: 0.2987 Epoch 7/50 3120/3120 [==============================] - 16s 5ms/step - loss: 2.0969 - acc: 0.2901 - val_loss: 2.0862 - val_acc: 0.2987 Epoch 8/50 3120/3120 [==============================] - 15s 5ms/step - loss: 2.0655 - acc: 0.2981 - val_loss: 2.0410 - val_acc: 0.3038 Epoch 9/50 3120/3120 [==============================] - 15s 5ms/step - loss: 2.0373 - acc: 0.3163 - val_loss: 2.0602 - val_acc: 0.3038 Epoch 10/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.9916 - acc: 0.3212 - val_loss: 2.0042 - val_acc: 0.3205 Epoch 11/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.9531 - acc: 0.3369 - val_loss: 1.9947 - val_acc: 0.3154 Epoch 12/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.9079 - acc: 0.3554 - val_loss: 2.0279 - val_acc: 0.3295 Epoch 13/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.8959 - acc: 0.3516 - val_loss: 2.0466 - val_acc: 0.3064 Epoch 14/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.8595 - acc: 0.3744 - val_loss: 1.9900 - val_acc: 0.3192 Epoch 15/50 3120/3120 [==============================] - 15s 5ms/step - loss: 1.8155 - acc: 0.3814 - val_loss: 1.9849 - val_acc: 0.3333 Epoch 16/50 3120/3120 [==============================] - 15s 5ms/step - loss: 1.7916 - acc: 0.4003 - val_loss: 2.0153 - val_acc: 0.3321 Epoch 17/50 3120/3120 [==============================] - 15s 5ms/step - loss: 1.7385 - acc: 0.4003 - val_loss: 1.9693 - val_acc: 0.3590 Epoch 18/50 3120/3120 [==============================] - 15s 5ms/step - loss: 1.7303 - acc: 0.4144 - val_loss: 1.9350 - val_acc: 0.3782 Epoch 19/50 3120/3120 [==============================] - 15s 5ms/step - loss: 1.7173 - acc: 0.4234 - val_loss: 1.9591 - val_acc: 0.3692 Epoch 20/50 3120/3120 [==============================] - 15s 5ms/step - loss: 1.6932 - acc: 0.4138 - val_loss: 1.9578 - val_acc: 0.3833 Epoch 21/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.6808 - acc: 0.4244 - val_loss: 1.9900 - val_acc: 0.3731 Epoch 22/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.6564 - acc: 0.4285 - val_loss: 1.9433 - val_acc: 0.3718 Epoch 23/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.5886 - acc: 0.4410 - val_loss: 2.0056 - val_acc: 0.3641 Epoch 24/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.5826 - acc: 0.4603 - val_loss: 1.9555 - val_acc: 0.3872 Epoch 25/50 3120/3120 [==============================] - 19s 6ms/step - loss: 1.5796 - acc: 0.4481 - val_loss: 1.9351 - val_acc: 0.3808 Epoch 26/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.5678 - acc: 0.4625 - val_loss: 1.9785 - val_acc: 0.3705 Epoch 27/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.5495 - acc: 0.4670 - val_loss: 1.9983 - val_acc: 0.3615 Epoch 28/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.5564 - acc: 0.4545 - val_loss: 2.0028 - val_acc: 0.3692 Epoch 29/50 3120/3120 [==============================] - 17s 6ms/step - loss: 1.5576 - acc: 0.4615 - val_loss: 2.0025 - val_acc: 0.3679 Epoch 30/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.4809 - acc: 0.4894 - val_loss: 1.9945 - val_acc: 0.3564 Epoch 31/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.4813 - acc: 0.4827 - val_loss: 2.0428 - val_acc: 0.3603 Epoch 32/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.5217 - acc: 0.4670 - val_loss: 2.0512 - val_acc: 0.3641 Epoch 33/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.4330 - acc: 0.4984 - val_loss: 2.0613 - val_acc: 0.3590 Epoch 34/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.4174 - acc: 0.5032 - val_loss: 2.0864 - val_acc: 0.3590 Epoch 35/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.4302 - acc: 0.5077 - val_loss: 2.0580 - val_acc: 0.3564 Epoch 36/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.4191 - acc: 0.5045 - val_loss: 2.0907 - val_acc: 0.3551 Epoch 37/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.4074 - acc: 0.5147 - val_loss: 2.1231 - val_acc: 0.3513 Epoch 38/50 3120/3120 [==============================] - 17s 5ms/step - loss: 1.4148 - acc: 0.5119 - val_loss: 2.1006 - val_acc: 0.3641 Epoch 39/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.4153 - acc: 0.5199 - val_loss: 2.1550 - val_acc: 0.3500 Epoch 40/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.3964 - acc: 0.5272 - val_loss: 2.1678 - val_acc: 0.3449 Epoch 41/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.3814 - acc: 0.5260 - val_loss: 2.1150 - val_acc: 0.3667 Epoch 42/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.3815 - acc: 0.5212 - val_loss: 2.0744 - val_acc: 0.3833 Epoch 43/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.3252 - acc: 0.5516 - val_loss: 2.0887 - val_acc: 0.3641 Epoch 44/50 3120/3120 [==============================] - 16s 5ms/step - loss: 1.2883 - acc: 0.5590 - val_loss: 2.1011 - val_acc: 0.3667 Epoch 45/50 320/3120 [==>...........................] - ETA: 13s - loss: 1.4344 - acc: 0.5000
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1157-216010866316> in <module>() 27 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 28 ---> 29 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=64, epochs=50, validation_split=0.2, shuffle=False) 30 31 tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
# New approach: we will use more conv layers but back to batch size 32. Fewer filters in each layer. Creates hierarchy
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[3120:], one_hot_encode(shuffled_y[3120:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 3120 samples, validate on 780 samples Epoch 1/50 3120/3120 [==============================] - 77s 25ms/step - loss: 2.5417 - acc: 0.0949 - val_loss: 2.4491 - val_acc: 0.1218 Epoch 2/50 3120/3120 [==============================] - 21s 7ms/step - loss: 2.4576 - acc: 0.1436 - val_loss: 2.3755 - val_acc: 0.1769 Epoch 3/50 3120/3120 [==============================] - 20s 6ms/step - loss: 2.3638 - acc: 0.1785 - val_loss: 2.3227 - val_acc: 0.2038 Epoch 4/50 3120/3120 [==============================] - 20s 6ms/step - loss: 2.2842 - acc: 0.2167 - val_loss: 2.2474 - val_acc: 0.2487 Epoch 5/50 3120/3120 [==============================] - 20s 7ms/step - loss: 2.1961 - acc: 0.2551 - val_loss: 2.2012 - val_acc: 0.2872 Epoch 6/50 3120/3120 [==============================] - 20s 7ms/step - loss: 2.1409 - acc: 0.2705 - val_loss: 2.1494 - val_acc: 0.2846 Epoch 7/50 3120/3120 [==============================] - 21s 7ms/step - loss: 2.0793 - acc: 0.2971 - val_loss: 2.1690 - val_acc: 0.2692 Epoch 8/50 3120/3120 [==============================] - 21s 7ms/step - loss: 2.0625 - acc: 0.3032 - val_loss: 2.1580 - val_acc: 0.2833 Epoch 9/50 3120/3120 [==============================] - 21s 7ms/step - loss: 2.0133 - acc: 0.3205 - val_loss: 2.1563 - val_acc: 0.2718 Epoch 10/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.9661 - acc: 0.3311 - val_loss: 2.1688 - val_acc: 0.2487 Epoch 11/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.9439 - acc: 0.3471 - val_loss: 2.1239 - val_acc: 0.2782 Epoch 12/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.8958 - acc: 0.3615 - val_loss: 2.2018 - val_acc: 0.2603 Epoch 13/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.8416 - acc: 0.3731 - val_loss: 2.1793 - val_acc: 0.2718 Epoch 14/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.8074 - acc: 0.3846 - val_loss: 2.1593 - val_acc: 0.3038 Epoch 15/50 3120/3120 [==============================] - 20s 6ms/step - loss: 1.7645 - acc: 0.3990 - val_loss: 2.1965 - val_acc: 0.2974 Epoch 16/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.7474 - acc: 0.4087 - val_loss: 2.2008 - val_acc: 0.3013 Epoch 17/50 3120/3120 [==============================] - 21s 7ms/step - loss: 1.7233 - acc: 0.4083 - val_loss: 2.1723 - val_acc: 0.3090 Epoch 18/50 3120/3120 [==============================] - 20s 7ms/step - loss: 1.6787 - acc: 0.4202 - val_loss: 2.1932 - val_acc: 0.3269 Epoch 19/50 2912/3120 [===========================>..] - ETA: 1s - loss: 1.5942 - acc: 0.4543
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1174-01bdda2df89c> in <module>() 26 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 27 ---> 28 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 29 30 tr_acc = second_iteration.evaluate(shuffled_data[:3120], one_hot_encode(shuffled_y[:3120]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2969 tensor_type = dtypes_module.as_dtype(tensor.dtype) 2970 array_vals.append(np.asarray(value, -> 2971 dtype=tensor_type.as_numpy_dtype)) 2972 2973 if self.feed_dict: C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order) 499 500 """ --> 501 return array(a, dtype, copy=False, order=order) 502 503 KeyboardInterrupt:
Given the small sizes of the images, the advice I was given to train on a smaller dataset can be "stretched". I would say, doubling the sizes of the datasets should rather positively affect performance. That is, 6240 training and 1560 validation. I have genuinely had it with the overfitting. I don't feel that reducing the complexity of the model can positively affect results because feature extraction is already not very strong as it is. I can add more layers but with this small a set it would probably just overfit even more. So I will double dataset size. But I do agree that training on 37500 samples is a very bad idea.
condensed_data = np.zeros((7800, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((7800), dtype=np.uint8)
ptrs = [600] * 13
training = 0
validation = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
if ptrs[val] <= 120:
condensed_data[6240 + validation] = classified_data_2[i]
condensed_y[6240 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((7800, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((7800), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(6240)]
shuffled_valid = [*range(6240, 7800)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(6240):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(1560):
shuffled_data[shuffled_valid[i]] = condensed_data[6240 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[6240 + i]
# New approach: we will use more conv layers but back to batch size 32. Fewer filters in each layer. Creates hierarchy
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[6240:], one_hot_encode(shuffled_y[6240:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 6240 samples, validate on 1560 samples Epoch 1/50 6240/6240 [==============================] - 85s 14ms/step - loss: 2.4723 - acc: 0.1239 - val_loss: 2.3794 - val_acc: 0.1724 Epoch 2/50 6240/6240 [==============================] - 28s 5ms/step - loss: 2.3041 - acc: 0.2090 - val_loss: 2.2303 - val_acc: 0.2385 Epoch 3/50 6240/6240 [==============================] - 28s 5ms/step - loss: 2.1778 - acc: 0.2696 - val_loss: 2.1098 - val_acc: 0.2994 Epoch 4/50 6240/6240 [==============================] - 26s 4ms/step - loss: 2.0985 - acc: 0.3013 - val_loss: 2.0406 - val_acc: 0.3365 Epoch 5/50 6240/6240 [==============================] - 26s 4ms/step - loss: 2.0245 - acc: 0.3276 - val_loss: 1.9962 - val_acc: 0.3468 Epoch 6/50 6240/6240 [==============================] - 26s 4ms/step - loss: 1.9653 - acc: 0.3391 - val_loss: 1.9621 - val_acc: 0.3571 Epoch 7/50 6240/6240 [==============================] - 28s 4ms/step - loss: 1.9211 - acc: 0.3579 - val_loss: 1.9733 - val_acc: 0.3442 Epoch 8/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.8630 - acc: 0.3704 - val_loss: 1.9490 - val_acc: 0.3635 Epoch 9/50 6240/6240 [==============================] - 28s 5ms/step - loss: 1.8258 - acc: 0.3859 - val_loss: 1.9405 - val_acc: 0.3583 Epoch 10/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.7958 - acc: 0.4021 - val_loss: 1.9660 - val_acc: 0.3654 Epoch 11/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.7534 - acc: 0.4131 - val_loss: 1.9423 - val_acc: 0.3763 Epoch 12/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.7079 - acc: 0.4274 - val_loss: 1.9899 - val_acc: 0.3667 Epoch 13/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.6890 - acc: 0.4239 - val_loss: 1.9981 - val_acc: 0.3718 Epoch 14/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.6651 - acc: 0.4389 - val_loss: 1.9603 - val_acc: 0.3750 Epoch 15/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.6323 - acc: 0.4471 - val_loss: 1.9735 - val_acc: 0.3821 Epoch 16/50 6240/6240 [==============================] - 63s 10ms/step - loss: 1.5832 - acc: 0.4622 - val_loss: 2.0434 - val_acc: 0.3756 Epoch 17/50 6240/6240 [==============================] - 29s 5ms/step - loss: 1.5357 - acc: 0.4835 - val_loss: 2.1322 - val_acc: 0.3571 Epoch 18/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.5309 - acc: 0.4864 - val_loss: 2.1269 - val_acc: 0.3622 Epoch 19/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.5261 - acc: 0.4880 - val_loss: 2.1543 - val_acc: 0.3609 Epoch 20/50 6240/6240 [==============================] - 29s 5ms/step - loss: 1.4722 - acc: 0.4965 - val_loss: 2.1644 - val_acc: 0.3513 Epoch 21/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.4630 - acc: 0.4968 - val_loss: 2.2363 - val_acc: 0.3423 Epoch 22/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.4492 - acc: 0.5038 - val_loss: 2.2729 - val_acc: 0.3321 Epoch 23/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.4370 - acc: 0.5079 - val_loss: 2.1977 - val_acc: 0.3583 Epoch 24/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.4186 - acc: 0.5111 - val_loss: 2.3163 - val_acc: 0.3526 Epoch 25/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.4137 - acc: 0.5104 - val_loss: 2.2149 - val_acc: 0.3667 Epoch 26/50 6240/6240 [==============================] - 27s 4ms/step - loss: 1.3987 - acc: 0.5181 - val_loss: 2.2015 - val_acc: 0.3673 Epoch 27/50 6240/6240 [==============================] - 23224s 4s/step - loss: 1.3655 - acc: 0.5216 - val_loss: 2.2988 - val_acc: 0.3641 Epoch 28/50 6240/6240 [==============================] - 54s 9ms/step - loss: 1.3340 - acc: 0.5377 - val_loss: 2.2855 - val_acc: 0.3513 Epoch 29/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.2864 - acc: 0.5494 - val_loss: 2.4211 - val_acc: 0.3494 Epoch 30/50 6240/6240 [==============================] - 45s 7ms/step - loss: 1.3276 - acc: 0.5412 - val_loss: 2.4184 - val_acc: 0.3449 Epoch 31/50 6240/6240 [==============================] - 44s 7ms/step - loss: 1.3153 - acc: 0.5359 - val_loss: 2.3546 - val_acc: 0.3410 Epoch 32/50 6240/6240 [==============================] - 42s 7ms/step - loss: 1.3024 - acc: 0.5415 - val_loss: 2.4365 - val_acc: 0.3526 Epoch 33/50 6240/6240 [==============================] - 42s 7ms/step - loss: 1.2759 - acc: 0.5519 - val_loss: 2.4131 - val_acc: 0.3538 Epoch 34/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.2071 - acc: 0.5731 - val_loss: 2.5478 - val_acc: 0.3410 Epoch 35/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.2150 - acc: 0.5713 - val_loss: 2.5851 - val_acc: 0.3513 Epoch 36/50 6240/6240 [==============================] - 44s 7ms/step - loss: 1.2031 - acc: 0.5756 - val_loss: 2.6353 - val_acc: 0.3506 Epoch 37/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.1962 - acc: 0.5675 - val_loss: 2.7816 - val_acc: 0.3474 Epoch 38/50 6240/6240 [==============================] - 42s 7ms/step - loss: 1.1936 - acc: 0.5774 - val_loss: 2.6291 - val_acc: 0.3436 Epoch 39/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.1748 - acc: 0.5756 - val_loss: 2.8563 - val_acc: 0.3353 Epoch 40/50 6240/6240 [==============================] - 42s 7ms/step - loss: 1.1696 - acc: 0.5809 - val_loss: 2.8196 - val_acc: 0.3391 Epoch 41/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.1593 - acc: 0.5851 - val_loss: 2.8718 - val_acc: 0.3378 Epoch 42/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.1630 - acc: 0.5739 - val_loss: 2.9468 - val_acc: 0.3314 Epoch 43/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.1589 - acc: 0.5848 - val_loss: 2.7754 - val_acc: 0.3256 Epoch 44/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.1535 - acc: 0.5918 - val_loss: 2.8154 - val_acc: 0.3442 Epoch 45/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.2150 - acc: 0.5655 - val_loss: 2.7708 - val_acc: 0.3385 Epoch 46/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.0993 - acc: 0.6099 - val_loss: 2.8992 - val_acc: 0.3308 Epoch 47/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.1248 - acc: 0.5968 - val_loss: 2.7804 - val_acc: 0.3436 Epoch 48/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.1339 - acc: 0.5966 - val_loss: 2.9573 - val_acc: 0.3513 Epoch 49/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.1410 - acc: 0.5878 - val_loss: 2.9805 - val_acc: 0.3449 Epoch 50/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.1317 - acc: 0.5894 - val_loss: 3.0930 - val_acc: 0.3442 Batch size 32: 65.5% (train) vs. 34.4% (validation)
'''class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 1559)
plt.imshow(shuffled_data[6240 + i][0])
l = second_iteration.predict(shuffled_data[6240 + i:6241 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[6240 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')''' # validation
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 6239)
plt.imshow(shuffled_data[i][0])
l = second_iteration.predict(shuffled_data[i:1 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}') # train
Expected: vehicles Got: vehicles Confidence: 0.8211724758148193 Second option: electronics Second confidence: 0.1554694026708603 Index: 565
One observation I just made is that there are some grayscale images which could unjustly sway the opinion of the CNN. How prevalent are these images actually?
gray = [0] * 15 # back to the old classes for a second
for i in range(37500):
if i % 100 == 0: print(i, end=" ")
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: gray[int(y_data[i])] += 1
gray
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500 15600 15700 15800 15900 16000 16100 16200 16300 16400 16500 16600 16700 16800 16900 17000 17100 17200 17300 17400 17500 17600 17700 17800 17900 18000 18100 18200 18300 18400 18500 18600 18700 18800 18900 19000 19100 19200 19300 19400 19500 19600 19700 19800 19900 20000 20100 20200 20300 20400 20500 20600 20700 20800 20900 21000 21100 21200 21300 21400 21500 21600 21700 21800 21900 22000 22100 22200 22300 22400 22500 22600 22700 22800 22900 23000 23100 23200 23300 23400 23500 23600 23700 23800 23900 24000 24100 24200 24300 24400 24500 24600 24700 24800 24900 25000 25100 25200 25300 25400 25500 25600 25700 25800 25900 26000 26100 26200 26300 26400 26500 26600 26700 26800 26900 27000 27100 27200 27300 27400 27500 27600 27700 27800 27900 28000 28100 28200 28300 28400 28500 28600 28700 28800 28900 29000 29100 29200 29300 29400 29500 29600 29700 29800 29900 30000 30100 30200 30300 30400 30500 30600 30700 30800 30900 31000 31100 31200 31300 31400 31500 31600 31700 31800 31900 32000 32100 32200 32300 32400 32500 32600 32700 32800 32900 33000 33100 33200 33300 33400 33500 33600 33700 33800 33900 34000 34100 34200 34300 34400 34500 34600 34700 34800 34900 35000 35100 35200 35300 35400 35500 35600 35700 35800 35900 36000 36100 36200 36300 36400 36500 36600 36700 36800 36900 37000 37100 37200 37300 37400
[17, 6, 28, 7, 69, 20, 23, 14, 21, 20, 47, 117, 19, 17, 67]
There aren't that many grayscale images, but it does little harm to remove them from the dataset, because the dataset size of 6240 train / 1560 valid is small enough that these grayscale images could contribute to the overfitting.
While we're at it, let's also remove those images, that have a white background. Our goal is to keep the images that were taken with a lot of background noise, which theoretically is a bad thing but since there are so few images without background noise (that there would be overfitting unless we severely reduced the model's capacity in which case there's a lack of learning) it should be the right thing to do.
condensed_data = np.zeros((7800, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((7800), dtype=np.uint8)
ptrs = [600] * 13
training = 0
validation = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: continue
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][0][k][l][0],classified_data_2[i][0][k][l][1],classified_data_2[i][0][k][l][2],\
classified_data_2[i][0][k+1][l][0],classified_data_2[i][0][k+1][l][1],classified_data_2[i][0][k+1][l][2],\
classified_data_2[i][0][k][l+1][0],classified_data_2[i][0][k][l+1][1],classified_data_2[i][0][k][l+1][2],\
classified_data_2[i][0][k+1][l+1][0],classified_data_2[i][0][k+1][l+1][1],classified_data_2[i][0][k+1][l+1][2])==255:
found = True
break
if found: break
if found: continue
if (training + validation) % 100 == 0: print(training + validation, end=" ")
if ptrs[val] <= 120:
condensed_data[6240 + validation] = classified_data_2[i]
condensed_y[6240 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((7800, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((7800), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(6240)]
shuffled_valid = [*range(6240, 7800)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(6240):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(1560):
shuffled_data[shuffled_valid[i]] = condensed_data[6240 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[6240 + i]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700
# New approach: we will use more conv layers but back to batch size 32. Fewer filters in each layer. Creates hierarchy
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[6240:], one_hot_encode(shuffled_y[6240:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 6240 samples, validate on 1560 samples Epoch 1/50 6240/6240 [==============================] - 100s 16ms/step - loss: 2.4542 - acc: 0.1357 - val_loss: 2.2775 - val_acc: 0.2192 Epoch 2/50 6240/6240 [==============================] - 43s 7ms/step - loss: 2.2563 - acc: 0.2402 - val_loss: 2.1731 - val_acc: 0.2500 Epoch 3/50 6240/6240 [==============================] - 45s 7ms/step - loss: 2.1321 - acc: 0.2766 - val_loss: 2.0995 - val_acc: 0.2917 Epoch 4/50 6240/6240 [==============================] - 42s 7ms/step - loss: 2.0492 - acc: 0.3178 - val_loss: 2.0487 - val_acc: 0.3090 Epoch 5/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.9894 - acc: 0.3341 - val_loss: 2.0032 - val_acc: 0.3218 Epoch 6/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.9401 - acc: 0.3442 - val_loss: 2.0005 - val_acc: 0.3321 Epoch 7/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.9167 - acc: 0.3519 - val_loss: 1.9960 - val_acc: 0.3397 Epoch 8/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.8748 - acc: 0.3707 - val_loss: 1.9555 - val_acc: 0.3404 Epoch 9/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.8401 - acc: 0.3821 - val_loss: 1.9944 - val_acc: 0.3429 Epoch 10/50 6240/6240 [==============================] - 45s 7ms/step - loss: 1.7999 - acc: 0.3910 - val_loss: 1.9647 - val_acc: 0.3462 Epoch 11/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.7485 - acc: 0.4082 - val_loss: 1.9703 - val_acc: 0.3564 Epoch 12/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.6998 - acc: 0.4272 - val_loss: 1.9567 - val_acc: 0.3526 Epoch 13/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.6901 - acc: 0.4317 - val_loss: 1.9742 - val_acc: 0.3513 Epoch 14/50 6240/6240 [==============================] - 39s 6ms/step - loss: 1.6544 - acc: 0.4511 - val_loss: 1.9724 - val_acc: 0.3558 Epoch 15/50 6240/6240 [==============================] - 39s 6ms/step - loss: 1.6343 - acc: 0.4510 - val_loss: 1.9821 - val_acc: 0.3494 Epoch 16/50 6240/6240 [==============================] - 39s 6ms/step - loss: 1.6105 - acc: 0.4558 - val_loss: 2.0424 - val_acc: 0.3353 Epoch 17/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.6014 - acc: 0.4572 - val_loss: 2.0743 - val_acc: 0.3474 Epoch 18/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.5654 - acc: 0.4657 - val_loss: 2.0529 - val_acc: 0.3474 Epoch 19/50 6240/6240 [==============================] - 43s 7ms/step - loss: 1.5375 - acc: 0.4697 - val_loss: 2.0308 - val_acc: 0.3571 Epoch 20/50 6240/6240 [==============================] - 41s 7ms/step - loss: 1.5334 - acc: 0.4774 - val_loss: 2.0639 - val_acc: 0.3532 Epoch 21/50 4256/6240 [===================>..........] - ETA: 12s - loss: 1.5066 - acc: 0.4777
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1419-bbf2564f0b93> in <module>() 26 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 27 ---> 28 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 29 30 tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
It didn't help. (Why should it have?)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[6240:], one_hot_encode(shuffled_y[6240:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 6240 samples, validate on 1560 samples Epoch 1/50 6240/6240 [==============================] - 128s 21ms/step - loss: 2.5072 - acc: 0.1236 - val_loss: 2.4201 - val_acc: 0.1526 Epoch 2/50 6240/6240 [==============================] - 83s 13ms/step - loss: 2.4339 - acc: 0.1562 - val_loss: 2.3172 - val_acc: 0.2006 Epoch 3/50 6240/6240 [==============================] - 86s 14ms/step - loss: 2.3552 - acc: 0.1772 - val_loss: 2.2917 - val_acc: 0.2032 Epoch 4/50 6240/6240 [==============================] - 85s 14ms/step - loss: 2.2990 - acc: 0.1965 - val_loss: 2.2161 - val_acc: 0.2417 Epoch 5/50 6240/6240 [==============================] - 83s 13ms/step - loss: 2.2521 - acc: 0.2196 - val_loss: 2.2844 - val_acc: 0.2429 Epoch 6/50 6240/6240 [==============================] - 84s 13ms/step - loss: 2.2207 - acc: 0.2365 - val_loss: 2.2014 - val_acc: 0.2545 Epoch 7/50 6240/6240 [==============================] - 85s 14ms/step - loss: 2.1647 - acc: 0.2505 - val_loss: 2.1686 - val_acc: 0.2692 Epoch 8/50 6240/6240 [==============================] - 84s 13ms/step - loss: 2.1236 - acc: 0.2686 - val_loss: 2.1057 - val_acc: 0.2865 Epoch 9/50 6240/6240 [==============================] - 84s 13ms/step - loss: 2.0549 - acc: 0.2913 - val_loss: 2.0706 - val_acc: 0.3205 Epoch 10/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.9964 - acc: 0.3138 - val_loss: 2.0636 - val_acc: 0.3199 Epoch 11/50 6240/6240 [==============================] - 83s 13ms/step - loss: 1.9429 - acc: 0.3245 - val_loss: 2.0779 - val_acc: 0.3256 Epoch 12/50 6240/6240 [==============================] - 83s 13ms/step - loss: 1.8840 - acc: 0.3447 - val_loss: 2.0728 - val_acc: 0.3173 Epoch 13/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.8415 - acc: 0.3598 - val_loss: 2.0811 - val_acc: 0.3353 Epoch 14/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.7809 - acc: 0.3739 - val_loss: 2.0486 - val_acc: 0.3538 Epoch 15/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.7501 - acc: 0.3861 - val_loss: 2.0949 - val_acc: 0.3417 Epoch 16/50 6240/6240 [==============================] - 83s 13ms/step - loss: 1.6933 - acc: 0.3998 - val_loss: 2.1419 - val_acc: 0.3147 Epoch 17/50 6240/6240 [==============================] - 83s 13ms/step - loss: 1.6338 - acc: 0.4168 - val_loss: 2.1585 - val_acc: 0.3340 Epoch 18/50 6240/6240 [==============================] - 86s 14ms/step - loss: 1.6082 - acc: 0.4324 - val_loss: 2.1690 - val_acc: 0.3192 Epoch 19/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.5742 - acc: 0.4362 - val_loss: 2.1747 - val_acc: 0.3404 Epoch 20/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.5326 - acc: 0.4546 - val_loss: 2.2374 - val_acc: 0.3391 Epoch 21/50 6240/6240 [==============================] - 84s 14ms/step - loss: 1.4993 - acc: 0.4625 - val_loss: 2.2630 - val_acc: 0.3263 Epoch 22/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.4796 - acc: 0.4686 - val_loss: 2.2428 - val_acc: 0.3295 Epoch 23/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.4198 - acc: 0.4905 - val_loss: 2.3719 - val_acc: 0.3391 Epoch 24/50 6240/6240 [==============================] - 85s 14ms/step - loss: 1.4040 - acc: 0.4942 - val_loss: 2.4073 - val_acc: 0.3468 Epoch 25/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.3370 - acc: 0.5165 - val_loss: 2.4659 - val_acc: 0.3462 Epoch 26/50 6240/6240 [==============================] - 84s 14ms/step - loss: 1.3098 - acc: 0.5292 - val_loss: 2.5165 - val_acc: 0.3564 Epoch 27/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.3048 - acc: 0.5285 - val_loss: 2.6713 - val_acc: 0.3449 Epoch 28/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.2427 - acc: 0.5481 - val_loss: 2.7710 - val_acc: 0.3423 Epoch 29/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.2426 - acc: 0.5429 - val_loss: 2.8339 - val_acc: 0.3301 Epoch 30/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.2023 - acc: 0.5609 - val_loss: 2.7550 - val_acc: 0.3487 Epoch 31/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.1878 - acc: 0.5641 - val_loss: 2.7745 - val_acc: 0.3474 Epoch 32/50 6240/6240 [==============================] - 86s 14ms/step - loss: 1.1850 - acc: 0.5633 - val_loss: 2.8126 - val_acc: 0.3327 Epoch 33/50 6240/6240 [==============================] - 85s 14ms/step - loss: 1.1565 - acc: 0.5769 - val_loss: 2.8859 - val_acc: 0.3372 Epoch 34/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.1083 - acc: 0.5989 - val_loss: 2.7397 - val_acc: 0.3353 Epoch 35/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.1173 - acc: 0.5829 - val_loss: 2.9334 - val_acc: 0.3147 Epoch 36/50 6240/6240 [==============================] - 84s 13ms/step - loss: 1.0686 - acc: 0.6056 - val_loss: 3.0379 - val_acc: 0.3256 Epoch 37/50 6240/6240 [==============================] - 89s 14ms/step - loss: 1.0991 - acc: 0.5973 - val_loss: 3.0699 - val_acc: 0.3173 Epoch 38/50 6240/6240 [==============================] - 86s 14ms/step - loss: 1.0943 - acc: 0.6059 - val_loss: 3.0266 - val_acc: 0.3442 Epoch 39/50 6240/6240 [==============================] - 84s 14ms/step - loss: 1.0892 - acc: 0.6045 - val_loss: 3.1112 - val_acc: 0.3372 Epoch 40/50 6240/6240 [==============================] - 85s 14ms/step - loss: 1.0750 - acc: 0.6077 - val_loss: 3.1165 - val_acc: 0.3423 Epoch 41/50 6240/6240 [==============================] - 86s 14ms/step - loss: 1.0525 - acc: 0.6173 - val_loss: 3.1496 - val_acc: 0.3397 Epoch 42/50 6240/6240 [==============================] - 85s 14ms/step - loss: 1.0119 - acc: 0.6276 - val_loss: 3.3197 - val_acc: 0.3301 Epoch 43/50 6240/6240 [==============================] - 85s 14ms/step - loss: 0.9722 - acc: 0.6425 - val_loss: 3.4304 - val_acc: 0.3372 Epoch 44/50 6240/6240 [==============================] - 85s 14ms/step - loss: 0.9832 - acc: 0.6428 - val_loss: 3.3578 - val_acc: 0.3404 Epoch 45/50 6240/6240 [==============================] - 85s 14ms/step - loss: 0.9723 - acc: 0.6444 - val_loss: 3.4693 - val_acc: 0.3327 Epoch 46/50 6240/6240 [==============================] - 86s 14ms/step - loss: 0.9624 - acc: 0.6357 - val_loss: 3.4497 - val_acc: 0.3263 Epoch 47/50 6240/6240 [==============================] - 85s 14ms/step - loss: 0.9736 - acc: 0.6442 - val_loss: 3.7026 - val_acc: 0.3295 Epoch 48/50 6240/6240 [==============================] - 85s 14ms/step - loss: 0.9246 - acc: 0.6665 - val_loss: 3.4127 - val_acc: 0.3269 Epoch 49/50 6240/6240 [==============================] - 85s 14ms/step - loss: 0.9135 - acc: 0.6628 - val_loss: 3.6338 - val_acc: 0.3340 Epoch 50/50 6240/6240 [==============================] - 84s 14ms/step - loss: 0.9338 - acc: 0.6601 - val_loss: 3.4291 - val_acc: 0.3308 Batch size 32: 73.2% (train) vs. 33.0% (validation)
https://www.dataquest.io/blog/regularization-in-machine-learning/
A weight-decay regularization approach may be necessary to ensure that the model learns properly (and not just on the training set).
Further investigation showed that the weights aren't even that high (or maybe they are and I just don't know what "high" means). Regardless, I just got the idea of reverting to the standard stochastic gradient descent optimizer and tuning its learning rate as a hyperparameter. And obviously reducing filters in earlier layers. The reasoning is that Adam adapts its learning rate which could lead it to adapt it for the purpose of memorizing the training set.
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=10, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.SGD(lr=0.01)
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[6240:], one_hot_encode(shuffled_y[6240:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 6240 samples, validate on 1560 samples Epoch 1/50 6240/6240 [==============================] - 70s 11ms/step - loss: 2.5234 - acc: 0.1090 - val_loss: 2.5448 - val_acc: 0.1199 Epoch 2/50 6240/6240 [==============================] - 29s 5ms/step - loss: 2.4324 - acc: 0.1639 - val_loss: 2.3833 - val_acc: 0.1891 Epoch 3/50 6240/6240 [==============================] - 28s 5ms/step - loss: 2.3549 - acc: 0.1944 - val_loss: 2.2795 - val_acc: 0.2224 Epoch 4/50 6240/6240 [==============================] - 27s 4ms/step - loss: 2.3185 - acc: 0.2024 - val_loss: 2.3023 - val_acc: 0.1968 Epoch 5/50 6240/6240 [==============================] - 28s 4ms/step - loss: 2.2935 - acc: 0.2179 - val_loss: 2.2612 - val_acc: 0.2115 Epoch 6/50 6240/6240 [==============================] - 28s 5ms/step - loss: 2.2608 - acc: 0.2221 - val_loss: 2.1948 - val_acc: 0.2532 Epoch 7/50 6240/6240 [==============================] - 29s 5ms/step - loss: 2.2423 - acc: 0.2338 - val_loss: 2.1755 - val_acc: 0.2622 Epoch 8/50 6240/6240 [==============================] - 28s 5ms/step - loss: 2.2102 - acc: 0.2463 - val_loss: 2.1668 - val_acc: 0.2673 Epoch 9/50 6240/6240 [==============================] - 28s 4ms/step - loss: 2.2030 - acc: 0.2473 - val_loss: 2.1579 - val_acc: 0.2673 Epoch 10/50 6240/6240 [==============================] - 28s 4ms/step - loss: 2.1806 - acc: 0.2625 - val_loss: 2.1494 - val_acc: 0.2641 Epoch 11/50 6240/6240 [==============================] - 27s 4ms/step - loss: 2.1666 - acc: 0.2700 - val_loss: 2.1262 - val_acc: 0.2744 Epoch 12/50 6240/6240 [==============================] - 27s 4ms/step - loss: 2.1246 - acc: 0.2756 - val_loss: 2.1158 - val_acc: 0.2827 Epoch 13/50 6240/6240 [==============================] - 27s 4ms/step - loss: 2.1287 - acc: 0.2678 - val_loss: 2.1659 - val_acc: 0.2801 Epoch 14/50 6240/6240 [==============================] - 27s 4ms/step - loss: 2.1140 - acc: 0.2801 - val_loss: 2.0953 - val_acc: 0.3000 Epoch 15/50 6240/6240 [==============================] - 31s 5ms/step - loss: 2.0825 - acc: 0.2981 - val_loss: 2.1176 - val_acc: 0.2962 Epoch 16/50 6240/6240 [==============================] - 32s 5ms/step - loss: 2.0694 - acc: 0.2958 - val_loss: 2.1335 - val_acc: 0.2994 Epoch 17/50 6240/6240 [==============================] - 30s 5ms/step - loss: 2.0602 - acc: 0.3011 - val_loss: 2.0706 - val_acc: 0.2974 Epoch 18/50 6240/6240 [==============================] - 28s 4ms/step - loss: 2.0315 - acc: 0.3069 - val_loss: 2.0549 - val_acc: 0.3224 Epoch 19/50 6240/6240 [==============================] - 29s 5ms/step - loss: 2.0087 - acc: 0.3176 - val_loss: 2.0838 - val_acc: 0.3109 Epoch 20/50 6240/6240 [==============================] - 30s 5ms/step - loss: 1.9896 - acc: 0.3245 - val_loss: 2.0460 - val_acc: 0.3096 Epoch 21/50 6240/6240 [==============================] - 30s 5ms/step - loss: 1.9778 - acc: 0.3240 - val_loss: 2.0470 - val_acc: 0.3186 Epoch 22/50 6240/6240 [==============================] - 31s 5ms/step - loss: 1.9485 - acc: 0.3413 - val_loss: 2.0598 - val_acc: 0.3212 Epoch 23/50 6240/6240 [==============================] - 31s 5ms/step - loss: 1.9275 - acc: 0.3373 - val_loss: 2.0455 - val_acc: 0.3250 Epoch 24/50 6240/6240 [==============================] - 31s 5ms/step - loss: 1.9071 - acc: 0.3546 - val_loss: 2.0564 - val_acc: 0.3269 Epoch 25/50 6240/6240 [==============================] - 30s 5ms/step - loss: 1.8978 - acc: 0.3521 - val_loss: 1.9962 - val_acc: 0.3346 Epoch 26/50 6240/6240 [==============================] - 29s 5ms/step - loss: 1.8769 - acc: 0.3593 - val_loss: 2.0556 - val_acc: 0.3385 Epoch 27/50 6240/6240 [==============================] - 31s 5ms/step - loss: 1.8530 - acc: 0.3683 - val_loss: 2.0316 - val_acc: 0.3340 Epoch 28/50 6240/6240 [==============================] - 29s 5ms/step - loss: 1.8597 - acc: 0.3670 - val_loss: 2.0009 - val_acc: 0.3449 Epoch 29/50 6240/6240 [==============================] - 28s 5ms/step - loss: 1.8295 - acc: 0.3788 - val_loss: 2.0590 - val_acc: 0.3391 Epoch 30/50 6240/6240 [==============================] - 29s 5ms/step - loss: 1.8213 - acc: 0.3838 - val_loss: 2.0807 - val_acc: 0.3327 Epoch 31/50 6240/6240 [==============================] - 28s 4ms/step - loss: 1.8124 - acc: 0.3904 - val_loss: 2.0404 - val_acc: 0.3333 Epoch 32/50 6240/6240 [==============================] - 29s 5ms/step - loss: 1.7857 - acc: 0.3920 - val_loss: 2.0673 - val_acc: 0.3410 Epoch 33/50 1344/6240 [=====>........................] - ETA: 20s - loss: 1.8328 - acc: 0.3571
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1430-4d5d3dc48c09> in <module>() 24 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 25 ---> 26 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 27 28 tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
It was a grave misconception.
Date: 26 November
I found this publicly available research paper at https://www.researchgate.net/publication/331677125_An_Overview_of_Overfitting_and_its_Solutions about overfitting. I would like to try the L2 regularization with my neural network, as opposed to L1, because this way features are still learned albeit with smaller weights.
Used documentation at https://keras.io/api/layers/regularizers/
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-5)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[6240:], one_hot_encode(shuffled_y[6240:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 6240 samples, validate on 1560 samples Epoch 1/50 6240/6240 [==============================] - 77s 12ms/step - loss: 2.4477 - acc: 0.1446 - val_loss: 2.3387 - val_acc: 0.1853 Epoch 2/50 6240/6240 [==============================] - 35s 6ms/step - loss: 2.3135 - acc: 0.2035 - val_loss: 2.2836 - val_acc: 0.2205 Epoch 3/50 6240/6240 [==============================] - 38s 6ms/step - loss: 2.2259 - acc: 0.2457 - val_loss: 2.1661 - val_acc: 0.2705 Epoch 4/50 6240/6240 [==============================] - 38s 6ms/step - loss: 2.1450 - acc: 0.2721 - val_loss: 2.1206 - val_acc: 0.2782 Epoch 5/50 6240/6240 [==============================] - 37s 6ms/step - loss: 2.0831 - acc: 0.2942 - val_loss: 2.1125 - val_acc: 0.2968 Epoch 6/50 6240/6240 [==============================] - 36s 6ms/step - loss: 2.0222 - acc: 0.3154 - val_loss: 2.0479 - val_acc: 0.3173 Epoch 7/50 6240/6240 [==============================] - 37s 6ms/step - loss: 1.9768 - acc: 0.3293 - val_loss: 2.0625 - val_acc: 0.3269 Epoch 8/50 6240/6240 [==============================] - 40s 6ms/step - loss: 1.9598 - acc: 0.3385 - val_loss: 2.0370 - val_acc: 0.3154 Epoch 9/50 6240/6240 [==============================] - 37s 6ms/step - loss: 1.8828 - acc: 0.3692 - val_loss: 2.0364 - val_acc: 0.3346 Epoch 10/50 6240/6240 [==============================] - 38s 6ms/step - loss: 1.8664 - acc: 0.3739 - val_loss: 2.0550 - val_acc: 0.3321 Epoch 11/50 6240/6240 [==============================] - 37s 6ms/step - loss: 1.8209 - acc: 0.3865 - val_loss: 1.9922 - val_acc: 0.3571 Epoch 12/50 6240/6240 [==============================] - 39s 6ms/step - loss: 1.7747 - acc: 0.4000 - val_loss: 2.0183 - val_acc: 0.3468 Epoch 13/50 6240/6240 [==============================] - 55s 9ms/step - loss: 1.7415 - acc: 0.4061 - val_loss: 2.0245 - val_acc: 0.3628 Epoch 14/50 6240/6240 [==============================] - 54s 9ms/step - loss: 1.7061 - acc: 0.4223 - val_loss: 2.0385 - val_acc: 0.3404 Epoch 15/50 6240/6240 [==============================] - 54s 9ms/step - loss: 1.6666 - acc: 0.4274 - val_loss: 2.0655 - val_acc: 0.3609 Epoch 16/50 6240/6240 [==============================] - 52s 8ms/step - loss: 1.6452 - acc: 0.4365 - val_loss: 2.0858 - val_acc: 0.3474 Epoch 17/50 6240/6240 [==============================] - 38s 6ms/step - loss: 1.6060 - acc: 0.4545 - val_loss: 2.0895 - val_acc: 0.3538 Epoch 18/50 6240/6240 [==============================] - 34s 5ms/step - loss: 1.5637 - acc: 0.4604 - val_loss: 2.1882 - val_acc: 0.3423 Epoch 19/50 6240/6240 [==============================] - 34s 5ms/step - loss: 1.5721 - acc: 0.4577 - val_loss: 2.1864 - val_acc: 0.3526 Epoch 20/50 6240/6240 [==============================] - 34s 5ms/step - loss: 1.5375 - acc: 0.4667 - val_loss: 2.1990 - val_acc: 0.3308 Epoch 21/50 6240/6240 [==============================] - 35s 6ms/step - loss: 1.5213 - acc: 0.4742 - val_loss: 2.2178 - val_acc: 0.3429 Epoch 22/50 6240/6240 [==============================] - 36s 6ms/step - loss: 1.4824 - acc: 0.4938 - val_loss: 2.3392 - val_acc: 0.3288 Epoch 23/50 6240/6240 [==============================] - 35s 6ms/step - loss: 1.4851 - acc: 0.4835 - val_loss: 2.2873 - val_acc: 0.3231 Epoch 24/50 6240/6240 [==============================] - 36s 6ms/step - loss: 1.4483 - acc: 0.5005 - val_loss: 2.3064 - val_acc: 0.3378 Epoch 25/50 6240/6240 [==============================] - 37s 6ms/step - loss: 1.4296 - acc: 0.5046 - val_loss: 2.3020 - val_acc: 0.3244 Epoch 26/50 6240/6240 [==============================] - 35s 6ms/step - loss: 1.4196 - acc: 0.5111 - val_loss: 2.2961 - val_acc: 0.3397 Epoch 27/50 6240/6240 [==============================] - 35s 6ms/step - loss: 1.4536 - acc: 0.5027 - val_loss: 2.3261 - val_acc: 0.3429 Epoch 28/50 6240/6240 [==============================] - 34s 6ms/step - loss: 1.4189 - acc: 0.5127 - val_loss: 2.3851 - val_acc: 0.3462 Epoch 29/50 6240/6240 [==============================] - 36s 6ms/step - loss: 1.4160 - acc: 0.5079 - val_loss: 2.5026 - val_acc: 0.3436 Epoch 30/50 3840/6240 [=================>............] - ETA: 13s - loss: 1.3737 - acc: 0.5211
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1436-ca80abfcb989> in <module>() 26 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 27 ---> 28 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 29 30 tr_acc = second_iteration.evaluate(shuffled_data[:6240], one_hot_encode(shuffled_y[:6240]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
condensed_data = np.zeros((15600, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((15600), dtype=np.uint8)
ptrs = [1200] * 13
training = 0
validation = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: continue
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][0][k][l][0],classified_data_2[i][0][k][l][1],classified_data_2[i][0][k][l][2],\
classified_data_2[i][0][k+1][l][0],classified_data_2[i][0][k+1][l][1],classified_data_2[i][0][k+1][l][2],\
classified_data_2[i][0][k][l+1][0],classified_data_2[i][0][k][l+1][1],classified_data_2[i][0][k][l+1][2],\
classified_data_2[i][0][k+1][l+1][0],classified_data_2[i][0][k+1][l+1][1],classified_data_2[i][0][k+1][l+1][2])==255:
found = True
break
if found: break
if found: continue
if (training + validation) % 100 == 0: print(training + validation, end=" ")
if ptrs[val] <= 240:
condensed_data[12480 + validation] = classified_data_2[i]
condensed_y[12480 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((15600, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((15600), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(12480)]
shuffled_valid = [*range(12480, 15600)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(12480):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(3120):
shuffled_data[shuffled_valid[i]] = condensed_data[12480 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[12480 + i]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-5)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 97s 8ms/step - loss: 2.4120 - acc: 0.1636 - val_loss: 2.1855 - val_acc: 0.2513 Epoch 2/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.1962 - acc: 0.2450 - val_loss: 2.0818 - val_acc: 0.3090 Epoch 3/50 12480/12480 [==============================] - 69s 6ms/step - loss: 2.0973 - acc: 0.2837 - val_loss: 2.0568 - val_acc: 0.3263 Epoch 4/50 12480/12480 [==============================] - 69s 5ms/step - loss: 2.0082 - acc: 0.3137 - val_loss: 1.9850 - val_acc: 0.3564 Epoch 5/50 12480/12480 [==============================] - 71s 6ms/step - loss: 1.9336 - acc: 0.3410 - val_loss: 1.9707 - val_acc: 0.3635 Epoch 6/50 12480/12480 [==============================] - 68s 5ms/step - loss: 1.8858 - acc: 0.3589 - val_loss: 1.9327 - val_acc: 0.3798 Epoch 7/50 12480/12480 [==============================] - 68s 5ms/step - loss: 1.8363 - acc: 0.3788 - val_loss: 1.9332 - val_acc: 0.3737 Epoch 8/50 12480/12480 [==============================] - 69s 6ms/step - loss: 1.7841 - acc: 0.3950 - val_loss: 1.9037 - val_acc: 0.3978 Epoch 9/50 12480/12480 [==============================] - 69s 6ms/step - loss: 1.7625 - acc: 0.3965 - val_loss: 1.9424 - val_acc: 0.3804 Epoch 10/50 12480/12480 [==============================] - 72s 6ms/step - loss: 1.7124 - acc: 0.4200 - val_loss: 1.9096 - val_acc: 0.4096 Epoch 11/50 12480/12480 [==============================] - 72s 6ms/step - loss: 1.6781 - acc: 0.4329 - val_loss: 1.8772 - val_acc: 0.4019 Epoch 12/50 12480/12480 [==============================] - 69s 6ms/step - loss: 1.6499 - acc: 0.4366 - val_loss: 1.9454 - val_acc: 0.3949 Epoch 13/50 12480/12480 [==============================] - 69s 6ms/step - loss: 1.6157 - acc: 0.4526 - val_loss: 1.9464 - val_acc: 0.3962 Epoch 14/50 12480/12480 [==============================] - 72s 6ms/step - loss: 1.5790 - acc: 0.4548 - val_loss: 1.9582 - val_acc: 0.4019 Epoch 15/50 12480/12480 [==============================] - 74s 6ms/step - loss: 1.5513 - acc: 0.4619 - val_loss: 1.9934 - val_acc: 0.4051 Epoch 16/50 7168/12480 [================>.............] - ETA: 28s - loss: 1.5157 - acc: 0.4805
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1442-56da06f639f1> in <module>() 26 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 27 ---> 28 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 29 30 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
why does it keep overfitting
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=8, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=48, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-5)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=13, input_dim=20, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-5)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 67s 5ms/step - loss: 2.4418 - acc: 0.1389 - val_loss: 2.3004 - val_acc: 0.2173 Epoch 2/50 12480/12480 [==============================] - 31s 3ms/step - loss: 2.3045 - acc: 0.2046 - val_loss: 2.2424 - val_acc: 0.2474 Epoch 3/50 12480/12480 [==============================] - 31s 2ms/step - loss: 2.2270 - acc: 0.2365 - val_loss: 2.1384 - val_acc: 0.2859 Epoch 4/50 12480/12480 [==============================] - 31s 2ms/step - loss: 2.1922 - acc: 0.2506 - val_loss: 2.0974 - val_acc: 0.3003 Epoch 5/50 12480/12480 [==============================] - 31s 2ms/step - loss: 2.1623 - acc: 0.2564 - val_loss: 2.0754 - val_acc: 0.3077 Epoch 6/50 12480/12480 [==============================] - 36s 3ms/step - loss: 2.1473 - acc: 0.2644 - val_loss: 2.1096 - val_acc: 0.3064 Epoch 7/50 12480/12480 [==============================] - 33s 3ms/step - loss: 2.1354 - acc: 0.2725 - val_loss: 2.0763 - val_acc: 0.3160 Epoch 8/50 12480/12480 [==============================] - 37s 3ms/step - loss: 2.1174 - acc: 0.2722 - val_loss: 2.0544 - val_acc: 0.3298 Epoch 9/50 12480/12480 [==============================] - 35s 3ms/step - loss: 2.0987 - acc: 0.2825 - val_loss: 2.0625 - val_acc: 0.3301 Epoch 10/50 12480/12480 [==============================] - 36s 3ms/step - loss: 2.0827 - acc: 0.2856 - val_loss: 2.0096 - val_acc: 0.3311 Epoch 11/50 12480/12480 [==============================] - 33s 3ms/step - loss: 2.0701 - acc: 0.2910 - val_loss: 2.0308 - val_acc: 0.3314 Epoch 12/50 12480/12480 [==============================] - 32s 3ms/step - loss: 2.0486 - acc: 0.2994 - val_loss: 2.0189 - val_acc: 0.3417 Epoch 13/50 12480/12480 [==============================] - 33s 3ms/step - loss: 2.0528 - acc: 0.2980 - val_loss: 2.0112 - val_acc: 0.3487 Epoch 14/50 12480/12480 [==============================] - 34s 3ms/step - loss: 2.0487 - acc: 0.2994 - val_loss: 1.9829 - val_acc: 0.3548 Epoch 15/50 12480/12480 [==============================] - 32s 3ms/step - loss: 2.0352 - acc: 0.3045 - val_loss: 2.0352 - val_acc: 0.3429 Epoch 16/50 12480/12480 [==============================] - 31s 2ms/step - loss: 2.0447 - acc: 0.3005 - val_loss: 2.0055 - val_acc: 0.3615 Epoch 17/50 12480/12480 [==============================] - 31s 2ms/step - loss: 2.0183 - acc: 0.3083 - val_loss: 1.9823 - val_acc: 0.3625 Epoch 18/50 12480/12480 [==============================] - 31s 2ms/step - loss: 2.0100 - acc: 0.3175 - val_loss: 2.0101 - val_acc: 0.3603 Epoch 19/50 9216/12480 [=====================>........] - ETA: 7s - loss: 2.0116 - acc: 0.3130
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1446-0fd4cf23e204> in <module>() 26 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 27 ---> 28 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 29 30 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
'''class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 1559)
plt.imshow(shuffled_data[6240 + i][0])
l = second_iteration.predict(shuffled_data[6240 + i:6241 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[6240 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')''' # validation
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 3129)
plt.imshow(shuffled_data[12480 + i][0])
l = second_iteration.predict(shuffled_data[12480 + i:12481 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[12480 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')
Expected: sea animals Got: vehicles Confidence: 0.25277429819107056 Second option: electronics Second confidence: 0.11957178264856339 Index: 605
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
# Run a grid search
for c in range(-3, 4):
r = 10**c
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(r)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(r)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(r)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=20, input_dim=48, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(r)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=16, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(r)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=30, validation_split=0.2, shuffle=False, verbose=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Regularization {r}:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Regularization 0.001: 54.2% (train) vs. 40.2% (validation) Regularization 0.01: 47.800000000000004% (train) vs. 41.400000000000006% (validation) Regularization 0.1: 22.700000000000003% (train) vs. 21.6% (validation) Regularization 1: 7.6000000000000005% (train) vs. 7.6000000000000005% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1484-7c20f8ee8329> in <module>() 29 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 30 ---> 31 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=30, validation_split=0.2, shuffle=False, verbose=False) 32 33 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 86s 7ms/step - loss: 2.8147 - acc: 0.1410 - val_loss: 2.4673 - val_acc: 0.2212 Epoch 2/50 12480/12480 [==============================] - 48s 4ms/step - loss: 2.4087 - acc: 0.2271 - val_loss: 2.2406 - val_acc: 0.2917 Epoch 3/50 12480/12480 [==============================] - 53s 4ms/step - loss: 2.2863 - acc: 0.2687 - val_loss: 2.1674 - val_acc: 0.3317 Epoch 4/50 12480/12480 [==============================] - 47s 4ms/step - loss: 2.2151 - acc: 0.2934 - val_loss: 2.0754 - val_acc: 0.3615 Epoch 5/50 12480/12480 [==============================] - 50s 4ms/step - loss: 2.1574 - acc: 0.3094 - val_loss: 2.1229 - val_acc: 0.3401 Epoch 6/50 12480/12480 [==============================] - 47s 4ms/step - loss: 2.1125 - acc: 0.3279 - val_loss: 2.0711 - val_acc: 0.3696 Epoch 7/50 12480/12480 [==============================] - 45s 4ms/step - loss: 2.0926 - acc: 0.3369 - val_loss: 2.0535 - val_acc: 0.3686 Epoch 8/50 12480/12480 [==============================] - 44s 4ms/step - loss: 2.0774 - acc: 0.3412 - val_loss: 1.9959 - val_acc: 0.3872 Epoch 9/50 12480/12480 [==============================] - 49s 4ms/step - loss: 2.0453 - acc: 0.3552 - val_loss: 2.0097 - val_acc: 0.3952 Epoch 10/50 12480/12480 [==============================] - 46s 4ms/step - loss: 2.0335 - acc: 0.3583 - val_loss: 2.0408 - val_acc: 0.3897 Epoch 11/50 12480/12480 [==============================] - 50s 4ms/step - loss: 2.0344 - acc: 0.3586 - val_loss: 2.0621 - val_acc: 0.3772 Epoch 12/50 12480/12480 [==============================] - 49s 4ms/step - loss: 2.0069 - acc: 0.3711 - val_loss: 2.0322 - val_acc: 0.3958 Epoch 13/50 12480/12480 [==============================] - 48s 4ms/step - loss: 2.0064 - acc: 0.3699 - val_loss: 2.0040 - val_acc: 0.3965 Epoch 14/50 12480/12480 [==============================] - 48s 4ms/step - loss: 1.9974 - acc: 0.3736 - val_loss: 2.0482 - val_acc: 0.3833 Epoch 15/50 12480/12480 [==============================] - 43s 3ms/step - loss: 1.9693 - acc: 0.3853 - val_loss: 2.0288 - val_acc: 0.4000 Epoch 16/50 12480/12480 [==============================] - 49s 4ms/step - loss: 1.9755 - acc: 0.3845 - val_loss: 1.9978 - val_acc: 0.4083 Epoch 17/50 11264/12480 [==========================>...] - ETA: 4s - loss: 1.9735 - acc: 0.3869
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1490-6729692b9efa> in <module>() 24 second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy']) 25 ---> 26 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 27 28 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
'''class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 1559)
plt.imshow(shuffled_data[6240 + i][0])
l = second_iteration.predict(shuffled_data[6240 + i:6241 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[6240 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')''' # validation
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 3129)
plt.imshow(shuffled_data[12480 + i][0])
l = second_iteration.predict(shuffled_data[12480 + i:12481 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[12480 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
if ind[0] != shuffled_y[12480 + i]: print(f'Confidence in the correct answer: {l[0][shuffled_y[12480 + i]]}')
print(f'Index: {i}')
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1775-3436f414f2b3> in <module>() 19 i = randint(0, 3129) 20 plt.imshow(shuffled_data[12480 + i][0]) ---> 21 l = second_iteration.predict(shuffled_data[12480 + i:12481 + i]) 22 ind = [j for j in range(13) if l[0][j] == max(l[0])] 23 second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in predict(self, x, batch_size, verbose, steps, max_queue_size, workers, use_multiprocessing) 1876 else: 1877 return training_arrays.predict_loop( -> 1878 self, x, batch_size=batch_size, verbose=verbose, steps=steps) 1879 1880 def train_on_batch(self, x, y=None, sample_weight=None, class_weight=None): C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in predict_loop(model, inputs, batch_size, verbose, steps) 324 ins_batch[i] = ins_batch[i].toarray() 325 --> 326 batch_outs = f(ins_batch) 327 if not isinstance(batch_outs, list): 328 batch_outs = [batch_outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2981 feed_symbols != self._feed_symbols or self.fetches != self._fetches or 2982 session != self._session): -> 2983 self._make_callable(feed_arrays, feed_symbols, symbol_vals, session) 2984 2985 fetched = self._callable_fn(*array_vals, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in _make_callable(self, feed_arrays, feed_symbols, symbol_vals, session) 2926 callable_opts.run_options.CopyFrom(self.run_options) 2927 # Create callable. -> 2928 callable_fn = session._make_callable_from_options(callable_opts) 2929 # Cache parameters corresponding to the generated callable, so that 2930 # we can detect future mismatches and refresh the callable. C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in _make_callable_from_options(self, callable_options) 1468 A handle to the new callable. 1469 """ -> 1470 self._extend_graph() 1471 return BaseSession._Callable(self, callable_options) 1472 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in _extend_graph(self) 1350 def _extend_graph(self): 1351 with self._graph._session_run_lock(): # pylint: disable=protected-access -> 1352 tf_session.ExtendSession(self._session) 1353 1354 # The threshold to run garbage collection to delete dead tensors. KeyboardInterrupt:
Observations, round 2
There are certain pairs of classes that closely relate to each other, e.g. fruits and vegetables vs. flowers; and it probably should be highlighted to the model that these two classes are different. Something similar was previously happening with land animals, but it was always going to be hard for the model to distinguish between omnivores and carnivores. (how do you even do that?) But fruits and vegetables vs. flowers is not particularly hard to distinguish. Same goes with furniture vs. electronics. I will therefore experiment with the idea of adding additional penalty terms to the loss function for these common "50/50" choices. (trees vs. insects is another 50/50 choice although it's less pronounced.)
It is important to note that for some of these 50/50s, the model only predicts one of them wrongly. So for instance, trees vs. insects: the model almsot never predicts trees on insects, but vice versa is very common. Specifications:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, cce) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 83s 7ms/step - loss: 2.9504 - acc: 0.1366 - val_loss: 2.6518 - val_acc: 0.1833 Epoch 2/50 12480/12480 [==============================] - 50s 4ms/step - loss: 2.4969 - acc: 0.2074 - val_loss: 2.3151 - val_acc: 0.2689 Epoch 3/50 12480/12480 [==============================] - 48s 4ms/step - loss: 2.3205 - acc: 0.2670 - val_loss: 2.2091 - val_acc: 0.3099 Epoch 4/50 12480/12480 [==============================] - 47s 4ms/step - loss: 2.2101 - acc: 0.2981 - val_loss: 2.1532 - val_acc: 0.3362 Epoch 5/50 12480/12480 [==============================] - 45s 4ms/step - loss: 2.1514 - acc: 0.3213 - val_loss: 2.1133 - val_acc: 0.3526 Epoch 6/50 12480/12480 [==============================] - 46s 4ms/step - loss: 2.1118 - acc: 0.3377 - val_loss: 2.0586 - val_acc: 0.3737 Epoch 7/50 12480/12480 [==============================] - 46s 4ms/step - loss: 2.0736 - acc: 0.3522 - val_loss: 2.0659 - val_acc: 0.3795 Epoch 8/50 12480/12480 [==============================] - 46s 4ms/step - loss: 2.0535 - acc: 0.3584 - val_loss: 2.0358 - val_acc: 0.3974 Epoch 9/50 12480/12480 [==============================] - 45s 4ms/step - loss: 2.0329 - acc: 0.3702 - val_loss: 2.0387 - val_acc: 0.4006 Epoch 10/50 12480/12480 [==============================] - 43s 3ms/step - loss: 2.0117 - acc: 0.3863 - val_loss: 2.0649 - val_acc: 0.3990 Epoch 11/50 12480/12480 [==============================] - 44s 4ms/step - loss: 2.0060 - acc: 0.3858 - val_loss: 2.0425 - val_acc: 0.4112 Epoch 12/50 12480/12480 [==============================] - 46s 4ms/step - loss: 1.9904 - acc: 0.3994 - val_loss: 2.0560 - val_acc: 0.4054 Epoch 13/50 352/12480 [..............................] - ETA: 40s - loss: 1.9829 - acc: 0.4006
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-1778-b261fb2ede39> in <module>() 31 second_iteration.compile(opt, scaled_loss, ['accuracy']) 32 ---> 33 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 34 35 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 12479)
plt.imshow(shuffled_data[i][0])
l = second_iteration.predict(shuffled_data[i:1 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')
'''class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 3129)
plt.imshow(shuffled_data[12480 + i][0])
l = second_iteration.predict(shuffled_data[12480 + i:12481 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[12480 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
if ind[0] != shuffled_y[12480 + i]: print(f'Confidence in the correct answer: {l[0][shuffled_y[12480 + i]]}')
print(f'Index: {i}')''';
Expected: insects Got: people Confidence: 0.17263589799404144 Second option: furniture Second confidence: 0.14831480383872986 Index: 6645
Observations, round 3
The main observation I have been able to make is that the model fails to detect very subtle images. Also there are a lot of "reptile" images of sea turtles and sea snakes and so on that could very easily be classified as sea animals (because of the blue background). Perhaps I should add this as a "50/50" pair for the loss function.
Indeed, adding a 1.2 scale factor to the loss function for the aforementioned 50/50 pairs has improved the accuracy slightly. Let's try more of that.
It seems as though the long-overdue EDA is finally coming now, with randomized analysis ;)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 87s 7ms/step - loss: 2.8538 - acc: 0.1414 - val_loss: 2.4724 - val_acc: 0.2202 Epoch 2/50 12480/12480 [==============================] - 52s 4ms/step - loss: 2.3820 - acc: 0.2340 - val_loss: 2.2185 - val_acc: 0.3170 Epoch 3/50 12480/12480 [==============================] - 45s 4ms/step - loss: 2.2390 - acc: 0.2751 - val_loss: 2.1901 - val_acc: 0.3314 Epoch 4/50 12480/12480 [==============================] - 52s 4ms/step - loss: 2.1678 - acc: 0.3003 - val_loss: 2.0934 - val_acc: 0.3574 Epoch 5/50 12480/12480 [==============================] - 53s 4ms/step - loss: 2.1160 - acc: 0.3234 - val_loss: 2.0502 - val_acc: 0.3718 Epoch 6/50 12480/12480 [==============================] - 53s 4ms/step - loss: 2.0912 - acc: 0.3394 - val_loss: 2.0593 - val_acc: 0.3795 Epoch 7/50 12480/12480 [==============================] - 52s 4ms/step - loss: 2.0645 - acc: 0.3522 - val_loss: 2.0665 - val_acc: 0.3888 Epoch 8/50 12480/12480 [==============================] - 55s 4ms/step - loss: 2.0475 - acc: 0.3654 - val_loss: 2.0091 - val_acc: 0.4054 Epoch 9/50 12480/12480 [==============================] - 51s 4ms/step - loss: 2.0137 - acc: 0.3770 - val_loss: 2.0153 - val_acc: 0.4058 Epoch 10/50 12480/12480 [==============================] - 51s 4ms/step - loss: 1.9889 - acc: 0.3903 - val_loss: 2.0088 - val_acc: 0.4058 Epoch 11/50 12480/12480 [==============================] - 52s 4ms/step - loss: 1.9806 - acc: 0.3951 - val_loss: 2.0139 - val_acc: 0.4112 Epoch 12/50 8384/12480 [===================>..........] - ETA: 17s - loss: 1.9543 - acc: 0.4029
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2103-217be9f981f5> in <module>() 31 second_iteration.compile(opt, scaled_loss, ['accuracy']) 32 ---> 33 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 34 35 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
Bruh I've been typing it in wrong the whole time ;) I genuinely don't know how it managed to improve.
We'll just have to wait and see I guess...
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 106s 9ms/step - loss: 2.9154 - acc: 0.1334 - val_loss: 2.5273 - val_acc: 0.2048 Epoch 2/50 12480/12480 [==============================] - 46s 4ms/step - loss: 2.4871 - acc: 0.2053 - val_loss: 2.3376 - val_acc: 0.2715 Epoch 3/50 12480/12480 [==============================] - 45s 4ms/step - loss: 2.3443 - acc: 0.2542 - val_loss: 2.2712 - val_acc: 0.2894 Epoch 4/50 12480/12480 [==============================] - 44s 4ms/step - loss: 2.2095 - acc: 0.2953 - val_loss: 2.1131 - val_acc: 0.3692 Epoch 5/50 12480/12480 [==============================] - 44s 4ms/step - loss: 2.1275 - acc: 0.3345 - val_loss: 2.0855 - val_acc: 0.3702 Epoch 6/50 12480/12480 [==============================] - 45s 4ms/step - loss: 2.0709 - acc: 0.3502 - val_loss: 2.0220 - val_acc: 0.4061 Epoch 7/50 12480/12480 [==============================] - 46s 4ms/step - loss: 2.0325 - acc: 0.3696 - val_loss: 2.0567 - val_acc: 0.3875 Epoch 8/50 12480/12480 [==============================] - 44s 4ms/step - loss: 1.9903 - acc: 0.3881 - val_loss: 2.0218 - val_acc: 0.4054 Epoch 9/50 12480/12480 [==============================] - 45s 4ms/step - loss: 1.9786 - acc: 0.3939 - val_loss: 2.0221 - val_acc: 0.4135 Epoch 10/50 12480/12480 [==============================] - 45s 4ms/step - loss: 1.9516 - acc: 0.4079 - val_loss: 1.9799 - val_acc: 0.4237 Epoch 11/50 12480/12480 [==============================] - 46s 4ms/step - loss: 1.9397 - acc: 0.4116 - val_loss: 1.9929 - val_acc: 0.4282 Epoch 12/50 12480/12480 [==============================] - 51s 4ms/step - loss: 1.9304 - acc: 0.4151 - val_loss: 2.0031 - val_acc: 0.4138 Epoch 13/50 416/12480 [>.............................] - ETA: 39s - loss: 1.8662 - acc: 0.4591
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2104-349f452aa856> in <module>() 31 second_iteration.compile(opt, scaled_loss, ['accuracy']) 32 ---> 33 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 34 35 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=30, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 106s 8ms/step - loss: 3.3143 - acc: 0.1503 - val_loss: 2.6641 - val_acc: 0.2497 Epoch 2/50 12480/12480 [==============================] - 77s 6ms/step - loss: 2.5049 - acc: 0.2278 - val_loss: 2.3239 - val_acc: 0.2920 Epoch 3/50 12480/12480 [==============================] - 78s 6ms/step - loss: 2.3128 - acc: 0.2750 - val_loss: 2.1575 - val_acc: 0.3426 Epoch 4/50 12480/12480 [==============================] - 81s 6ms/step - loss: 2.1876 - acc: 0.3212 - val_loss: 2.1176 - val_acc: 0.3769 Epoch 5/50 12480/12480 [==============================] - 80s 6ms/step - loss: 2.1367 - acc: 0.3447 - val_loss: 2.0647 - val_acc: 0.4022 Epoch 6/50 12480/12480 [==============================] - 80s 6ms/step - loss: 2.0905 - acc: 0.3642 - val_loss: 2.0763 - val_acc: 0.4013 Epoch 7/50 12480/12480 [==============================] - 75s 6ms/step - loss: 2.0577 - acc: 0.3813 - val_loss: 2.0475 - val_acc: 0.4064 Epoch 8/50 12480/12480 [==============================] - 76s 6ms/step - loss: 2.0314 - acc: 0.3821 - val_loss: 2.0374 - val_acc: 0.4144 Epoch 9/50 12480/12480 [==============================] - 72s 6ms/step - loss: 1.9845 - acc: 0.4115 - val_loss: 2.0435 - val_acc: 0.4218 Epoch 10/50 12480/12480 [==============================] - 73s 6ms/step - loss: 1.9667 - acc: 0.4161 - val_loss: 2.0069 - val_acc: 0.4253 Epoch 11/50 12480/12480 [==============================] - 70s 6ms/step - loss: 1.9588 - acc: 0.4179 - val_loss: 2.0484 - val_acc: 0.4292 Epoch 12/50 1376/12480 [==>...........................] - ETA: 57s - loss: 1.8816 - acc: 0.4564
KeyboardInterrupt
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=40, input_dim=192, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=30, input_dim=40, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-2)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 110s 9ms/step - loss: 3.3880 - acc: 0.1552 - val_loss: 2.6938 - val_acc: 0.2074 Epoch 2/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.5314 - acc: 0.2073 - val_loss: 2.3643 - val_acc: 0.2538 Epoch 3/50 12480/12480 [==============================] - 71s 6ms/step - loss: 2.3616 - acc: 0.2353 - val_loss: 2.2419 - val_acc: 0.2782 Epoch 4/50 12480/12480 [==============================] - 70s 6ms/step - loss: 2.3020 - acc: 0.2569 - val_loss: 2.1648 - val_acc: 0.3119 Epoch 5/50 12480/12480 [==============================] - 69s 6ms/step - loss: 2.2497 - acc: 0.2820 - val_loss: 2.1617 - val_acc: 0.3154 Epoch 6/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.2254 - acc: 0.2950 - val_loss: 2.1403 - val_acc: 0.3490 Epoch 7/50 12480/12480 [==============================] - 69s 6ms/step - loss: 2.1849 - acc: 0.3095 - val_loss: 2.1226 - val_acc: 0.3372 Epoch 8/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.1770 - acc: 0.3202 - val_loss: 2.1655 - val_acc: 0.3381 Epoch 9/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.1641 - acc: 0.3244 - val_loss: 2.1065 - val_acc: 0.3545 Epoch 10/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.1381 - acc: 0.3367 - val_loss: 2.0865 - val_acc: 0.3712 Epoch 11/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.1285 - acc: 0.3378 - val_loss: 2.0819 - val_acc: 0.3724 Epoch 12/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.1259 - acc: 0.3418 - val_loss: 2.0652 - val_acc: 0.3750 Epoch 13/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.1063 - acc: 0.3559 - val_loss: 2.0581 - val_acc: 0.3795 Epoch 14/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.0948 - acc: 0.3603 - val_loss: 2.0694 - val_acc: 0.3792 Epoch 15/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.0897 - acc: 0.3610 - val_loss: 2.0499 - val_acc: 0.3913 Epoch 16/50 12480/12480 [==============================] - 69s 5ms/step - loss: 2.0871 - acc: 0.3666 - val_loss: 2.0637 - val_acc: 0.3952 Epoch 17/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.0787 - acc: 0.3613 - val_loss: 2.0722 - val_acc: 0.3971 Epoch 18/50 12480/12480 [==============================] - 70s 6ms/step - loss: 2.0736 - acc: 0.3724 - val_loss: 2.0459 - val_acc: 0.4071 Epoch 19/50 12480/12480 [==============================] - 69s 5ms/step - loss: 2.0573 - acc: 0.3799 - val_loss: 2.0493 - val_acc: 0.4071 Epoch 20/50 12480/12480 [==============================] - 68s 5ms/step - loss: 2.0510 - acc: 0.3800 - val_loss: 2.0339 - val_acc: 0.4131 Epoch 21/50 12480/12480 [==============================] - 67s 5ms/step - loss: 2.0509 - acc: 0.3800 - val_loss: 2.0629 - val_acc: 0.4064 Epoch 22/50 12480/12480 [==============================] - 67s 5ms/step - loss: 2.0500 - acc: 0.3825 - val_loss: 2.0702 - val_acc: 0.4042 Epoch 23/50 9344/12480 [=====================>........] - ETA: 16s - loss: 2.0391 - acc: 0.3921
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2108-3d3a1ca23765> in <module>() 33 second_iteration.compile(opt, scaled_loss, ['accuracy']) 34 ---> 35 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 36 37 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=80, input_dim=256, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=40, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=40, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 133s 11ms/step - loss: 2.4636 - acc: 0.1591 - val_loss: 2.3068 - val_acc: 0.2269 Epoch 2/50 12480/12480 [==============================] - 82s 7ms/step - loss: 2.2625 - acc: 0.2377 - val_loss: 2.1969 - val_acc: 0.2917 Epoch 3/50 12480/12480 [==============================] - 82s 7ms/step - loss: 2.1857 - acc: 0.2793 - val_loss: 2.1244 - val_acc: 0.3244 Epoch 4/50 12480/12480 [==============================] - 83s 7ms/step - loss: 2.1230 - acc: 0.3040 - val_loss: 2.0904 - val_acc: 0.3417 Epoch 5/50 12480/12480 [==============================] - 73s 6ms/step - loss: 2.0754 - acc: 0.3197 - val_loss: 2.0455 - val_acc: 0.3551 Epoch 6/50 12480/12480 [==============================] - 73s 6ms/step - loss: 2.0316 - acc: 0.3470 - val_loss: 2.0341 - val_acc: 0.3686 Epoch 7/50 12480/12480 [==============================] - 68s 5ms/step - loss: 1.9840 - acc: 0.3619 - val_loss: 1.9975 - val_acc: 0.3821 Epoch 8/50 12480/12480 [==============================] - 68s 5ms/step - loss: 1.9478 - acc: 0.3788 - val_loss: 1.9716 - val_acc: 0.3929 Epoch 9/50 12480/12480 [==============================] - 67s 5ms/step - loss: 1.9289 - acc: 0.3849 - val_loss: 2.0119 - val_acc: 0.3827 Epoch 10/50 12480/12480 [==============================] - 68s 5ms/step - loss: 1.8845 - acc: 0.3983 - val_loss: 1.9700 - val_acc: 0.4080 Epoch 11/50 12480/12480 [==============================] - 71s 6ms/step - loss: 1.8676 - acc: 0.4062 - val_loss: 1.9956 - val_acc: 0.4026 Epoch 12/50 12480/12480 [==============================] - 69s 6ms/step - loss: 1.8507 - acc: 0.4135 - val_loss: 1.9568 - val_acc: 0.4176 Epoch 13/50 12480/12480 [==============================] - 67s 5ms/step - loss: 1.8175 - acc: 0.4280 - val_loss: 1.9817 - val_acc: 0.4125 Epoch 14/50 12480/12480 [==============================] - 68s 5ms/step - loss: 1.8228 - acc: 0.4273 - val_loss: 1.9694 - val_acc: 0.4026 Epoch 15/50 12480/12480 [==============================] - 67s 5ms/step - loss: 1.7903 - acc: 0.4356 - val_loss: 1.9594 - val_acc: 0.4189 Epoch 16/50 12480/12480 [==============================] - 67s 5ms/step - loss: 1.7780 - acc: 0.4413 - val_loss: 1.9713 - val_acc: 0.4272 Epoch 17/50 12480/12480 [==============================] - 69s 5ms/step - loss: 1.7703 - acc: 0.4434 - val_loss: 2.0016 - val_acc: 0.4106 Epoch 18/50 12480/12480 [==============================] - 67s 5ms/step - loss: 1.7494 - acc: 0.4548 - val_loss: 2.0153 - val_acc: 0.4304 Epoch 19/50 800/12480 [>.............................] - ETA: 1:00 - loss: 1.6906 - acc: 0.4750
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2135-b2cc9271036b> in <module>() 36 second_iteration.compile(opt, scaled_loss, ['accuracy']) 37 ---> 38 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 39 40 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=80, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=40, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=40, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[12480:], one_hot_encode(shuffled_y[12480:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 12480 samples, validate on 3120 samples Epoch 1/50 12480/12480 [==============================] - 227s 18ms/step - loss: 2.6080 - acc: 0.1534 - val_loss: 2.3551 - val_acc: 0.2353 Epoch 2/50 12480/12480 [==============================] - 153s 12ms/step - loss: 2.3345 - acc: 0.2268 - val_loss: 2.2282 - val_acc: 0.2846 Epoch 3/50 12480/12480 [==============================] - 140s 11ms/step - loss: 2.2363 - acc: 0.2661 - val_loss: 2.1824 - val_acc: 0.2984 Epoch 4/50 12480/12480 [==============================] - 135s 11ms/step - loss: 2.1850 - acc: 0.2919 - val_loss: 2.1274 - val_acc: 0.3247 Epoch 5/50 12480/12480 [==============================] - 136s 11ms/step - loss: 2.1298 - acc: 0.3141 - val_loss: 2.0827 - val_acc: 0.3487 Epoch 6/50 12480/12480 [==============================] - 140s 11ms/step - loss: 2.0909 - acc: 0.3267 - val_loss: 2.0589 - val_acc: 0.3513 Epoch 7/50 12480/12480 [==============================] - 180s 14ms/step - loss: 2.0495 - acc: 0.3414 - val_loss: 2.0652 - val_acc: 0.3606 Epoch 8/50 12480/12480 [==============================] - 214s 17ms/step - loss: 2.0128 - acc: 0.3606 - val_loss: 2.0490 - val_acc: 0.3670 Epoch 9/50 12480/12480 [==============================] - 218s 17ms/step - loss: 1.9773 - acc: 0.3762 - val_loss: 2.0357 - val_acc: 0.3763 Epoch 10/50 96/12480 [..............................] - ETA: 3:23 - loss: 1.8445 - acc: 0.4479
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2145-923e1e3fca93> in <module>() 36 second_iteration.compile(opt, scaled_loss, ['accuracy']) 37 ---> 38 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=50, validation_split=0.2, shuffle=False) 39 40 tr_acc = second_iteration.evaluate(shuffled_data[:12480], one_hot_encode(shuffled_y[:12480]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2945 raise TypeError('`inputs` should be a list or tuple.') 2946 -> 2947 session = get_session() 2948 feed_arrays = [] 2949 array_vals = [] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in get_session() 467 if not _MANUAL_VAR_INIT: 468 with session.graph.as_default(): --> 469 _initialize_variables(session) 470 return session 471 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in _initialize_variables(session) 722 variables = _get_variables(ops.get_default_graph()) 723 candidate_vars = [] --> 724 for v in variables: 725 if not getattr(v, '_keras_initialized', False): 726 candidate_vars.append(v) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\_weakrefset.py in __iter__(self) 63 # Caveat: the iterator will keep a strong reference to 64 # `item` until it is resumed or closed. ---> 65 yield item 66 67 def __len__(self): KeyboardInterrupt:
condensed_data = np.zeros((6500, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((6500), dtype=np.uint8)
ptrs = [500] * 13
training = 0
validation = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: continue
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][0][k][l][0],classified_data_2[i][0][k][l][1],classified_data_2[i][0][k][l][2],\
classified_data_2[i][0][k+1][l][0],classified_data_2[i][0][k+1][l][1],classified_data_2[i][0][k+1][l][2],\
classified_data_2[i][0][k][l+1][0],classified_data_2[i][0][k][l+1][1],classified_data_2[i][0][k][l+1][2],\
classified_data_2[i][0][k+1][l+1][0],classified_data_2[i][0][k+1][l+1][1],classified_data_2[i][0][k+1][l+1][2])==255:
found = True
break
if found: break
if found: continue
if (training + validation) % 100 == 0: print(training + validation, end=" ")
if ptrs[val] <= 100:
condensed_data[5200 + validation] = classified_data_2[i]
condensed_y[5200 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((6500, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((6500), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(5200)]
shuffled_valid = [*range(5200, 6500)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(5200):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(1300):
shuffled_data[shuffled_valid[i]] = condensed_data[5200 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[5200 + i]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=50, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=40, input_dim=50, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=40, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=25, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 5200 samples, validate on 1300 samples Epoch 1/25 5200/5200 [==============================] - 121s 23ms/step - loss: 2.6483 - acc: 0.0917 - val_loss: 2.4840 - val_acc: 0.1392 Epoch 2/25 5200/5200 [==============================] - 64s 12ms/step - loss: 2.4729 - acc: 0.1658 - val_loss: 2.3221 - val_acc: 0.2038 Epoch 3/25 5200/5200 [==============================] - 64s 12ms/step - loss: 2.3765 - acc: 0.1940 - val_loss: 2.2613 - val_acc: 0.2323 Epoch 4/25 5200/5200 [==============================] - 64s 12ms/step - loss: 2.3142 - acc: 0.2244 - val_loss: 2.2191 - val_acc: 0.2508 Epoch 5/25 5200/5200 [==============================] - 70s 13ms/step - loss: 2.2761 - acc: 0.2335 - val_loss: 2.1940 - val_acc: 0.2454 Epoch 6/25 5200/5200 [==============================] - 63s 12ms/step - loss: 2.2468 - acc: 0.2427 - val_loss: 2.1838 - val_acc: 0.2638 Epoch 7/25 5200/5200 [==============================] - 64s 12ms/step - loss: 2.2202 - acc: 0.2562 - val_loss: 2.1562 - val_acc: 0.2815 Epoch 8/25 5200/5200 [==============================] - 64s 12ms/step - loss: 2.1910 - acc: 0.2794 - val_loss: 2.1145 - val_acc: 0.2931 Epoch 9/25 5200/5200 [==============================] - 70s 13ms/step - loss: 2.1632 - acc: 0.2856 - val_loss: 2.1038 - val_acc: 0.3100 Epoch 10/25 5200/5200 [==============================] - 74s 14ms/step - loss: 2.1359 - acc: 0.3085 - val_loss: 2.1041 - val_acc: 0.2977 Epoch 11/25 5200/5200 [==============================] - 73s 14ms/step - loss: 2.0928 - acc: 0.3254 - val_loss: 2.0731 - val_acc: 0.3262 Epoch 12/25 5200/5200 [==============================] - 72s 14ms/step - loss: 2.0629 - acc: 0.3279 - val_loss: 2.0758 - val_acc: 0.3438 Epoch 13/25 5200/5200 [==============================] - 73s 14ms/step - loss: 2.0415 - acc: 0.3358 - val_loss: 2.0863 - val_acc: 0.3462 Epoch 14/25 5200/5200 [==============================] - 69s 13ms/step - loss: 2.0107 - acc: 0.3587 - val_loss: 2.0843 - val_acc: 0.3500 Epoch 15/25 5200/5200 [==============================] - 63s 12ms/step - loss: 1.9805 - acc: 0.3644 - val_loss: 2.0622 - val_acc: 0.3577 Epoch 16/25 5200/5200 [==============================] - 64s 12ms/step - loss: 1.9835 - acc: 0.3675 - val_loss: 2.0470 - val_acc: 0.3500 Epoch 17/25 416/5200 [=>............................] - ETA: 53s - loss: 2.0034 - acc: 0.3389
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2152-f92baf320104> in <module>() 36 second_iteration.compile(opt, scaled_loss, ['accuracy']) 37 ---> 38 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=25, validation_split=0.2, shuffle=False) 39 40 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2945 raise TypeError('`inputs` should be a list or tuple.') 2946 -> 2947 session = get_session() 2948 feed_arrays = [] 2949 array_vals = [] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in get_session() 467 if not _MANUAL_VAR_INIT: 468 with session.graph.as_default(): --> 469 _initialize_variables(session) 470 return session 471 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in _initialize_variables(session) 723 candidate_vars = [] 724 for v in variables: --> 725 if not getattr(v, '_keras_initialized', False): 726 candidate_vars.append(v) 727 if candidate_vars: KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.25)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(3e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=60, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=60, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=35, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 5200 samples, validate on 1300 samples Epoch 1/35 5200/5200 [==============================] - 223s 43ms/step - loss: 3.7597 - acc: 0.1413 - val_loss: 3.0044 - val_acc: 0.1962 Epoch 2/35 5200/5200 [==============================] - 145s 28ms/step - loss: 2.9046 - acc: 0.1938 - val_loss: 2.7186 - val_acc: 0.2200 Epoch 3/35 5200/5200 [==============================] - 146s 28ms/step - loss: 2.7152 - acc: 0.2221 - val_loss: 2.5587 - val_acc: 0.2569 Epoch 4/35 5200/5200 [==============================] - 146s 28ms/step - loss: 2.5334 - acc: 0.2688 - val_loss: 2.4636 - val_acc: 0.2731 Epoch 5/35 5200/5200 [==============================] - 148s 28ms/step - loss: 2.3968 - acc: 0.2942 - val_loss: 2.3390 - val_acc: 0.3362 Epoch 6/35 5200/5200 [==============================] - 152s 29ms/step - loss: 2.3032 - acc: 0.3190 - val_loss: 2.2713 - val_acc: 0.3385 Epoch 7/35 5200/5200 [==============================] - 150s 29ms/step - loss: 2.2196 - acc: 0.3419 - val_loss: 2.2111 - val_acc: 0.3477 Epoch 8/35 5200/5200 [==============================] - 162s 31ms/step - loss: 2.1505 - acc: 0.3579 - val_loss: 2.1946 - val_acc: 0.3585 Epoch 9/35 5200/5200 [==============================] - 162s 31ms/step - loss: 2.0798 - acc: 0.3746 - val_loss: 2.1143 - val_acc: 0.3838 Epoch 10/35 5200/5200 [==============================] - 163s 31ms/step - loss: 2.0334 - acc: 0.3938 - val_loss: 2.1078 - val_acc: 0.3862 Epoch 11/35 5200/5200 [==============================] - 146s 28ms/step - loss: 1.9557 - acc: 0.4138 - val_loss: 2.0885 - val_acc: 0.3992 Epoch 12/35 5200/5200 [==============================] - 149s 29ms/step - loss: 1.9309 - acc: 0.4200 - val_loss: 2.1061 - val_acc: 0.3977 Epoch 13/35 1696/5200 [========>.....................] - ETA: 1:35 - loss: 1.8890 - acc: 0.4287
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2165-3a122a44f273> in <module>() 36 second_iteration.compile(opt, scaled_loss, ['accuracy']) 37 ---> 38 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=35, validation_split=0.2, shuffle=False) 39 40 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
Combating the issue requires more regularization. Also I will add a hidden layer.
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(5e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(5e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(5e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(5e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(5e-3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(5e-3)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=80, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=40, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=13, input_dim=40, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(1e-3)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=35, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 5200 samples, validate on 1300 samples Epoch 1/35 5200/5200 [==============================] - 221s 42ms/step - loss: 4.7824 - acc: 0.1121 - val_loss: 3.2069 - val_acc: 0.1754 Epoch 2/35 5200/5200 [==============================] - 159s 31ms/step - loss: 2.9881 - acc: 0.1758 - val_loss: 2.7522 - val_acc: 0.1831 Epoch 3/35 5184/5200 [============================>.] - ETA: 0s - loss: 2.6630 - acc: 0.2068
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=80, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=60, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dense(units=13, input_dim=60, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 5200 samples, validate on 1300 samples Epoch 1/40 5200/5200 [==============================] - 242s 47ms/step - loss: 2.4168 - acc: 0.1644 - val_loss: 2.2326 - val_acc: 0.2123 Epoch 2/40 5200/5200 [==============================] - 139s 27ms/step - loss: 2.2498 - acc: 0.2252 - val_loss: 2.1946 - val_acc: 0.2508 Epoch 3/40 5200/5200 [==============================] - 142s 27ms/step - loss: 2.1826 - acc: 0.2573 - val_loss: 2.0822 - val_acc: 0.2846 Epoch 4/40 5200/5200 [==============================] - 145s 28ms/step - loss: 2.0638 - acc: 0.3069 - val_loss: 2.0119 - val_acc: 0.3308 Epoch 5/40 5200/5200 [==============================] - 143s 27ms/step - loss: 1.9574 - acc: 0.3446 - val_loss: 2.0555 - val_acc: 0.3285 Epoch 6/40 5200/5200 [==============================] - 144s 28ms/step - loss: 1.8356 - acc: 0.3810 - val_loss: 2.0402 - val_acc: 0.3577 Epoch 7/40 5200/5200 [==============================] - 154s 30ms/step - loss: 1.7421 - acc: 0.4212 - val_loss: 1.9836 - val_acc: 0.3638 Epoch 8/40 5200/5200 [==============================] - 150s 29ms/step - loss: 1.6660 - acc: 0.4456 - val_loss: 1.9507 - val_acc: 0.3954 Epoch 9/40 5200/5200 [==============================] - 150s 29ms/step - loss: 1.5405 - acc: 0.4860 - val_loss: 1.9824 - val_acc: 0.3877 Epoch 10/40 5200/5200 [==============================] - 150s 29ms/step - loss: 1.4731 - acc: 0.5073 - val_loss: 2.1821 - val_acc: 0.3508 Epoch 11/40 5200/5200 [==============================] - 149s 29ms/step - loss: 1.3913 - acc: 0.5279 - val_loss: 2.2359 - val_acc: 0.3731 Epoch 12/40 5200/5200 [==============================] - 152s 29ms/step - loss: 1.3407 - acc: 0.5538 - val_loss: 2.2798 - val_acc: 0.3377 Epoch 13/40 5200/5200 [==============================] - 157s 30ms/step - loss: 1.2559 - acc: 0.5812 - val_loss: 2.3641 - val_acc: 0.3454 Epoch 14/40 5200/5200 [==============================] - 193s 37ms/step - loss: 1.2365 - acc: 0.5858 - val_loss: 2.2838 - val_acc: 0.3592 Epoch 15/40 5200/5200 [==============================] - 158s 30ms/step - loss: 1.1459 - acc: 0.6229 - val_loss: 2.4503 - val_acc: 0.3477 Epoch 16/40 5200/5200 [==============================] - 151s 29ms/step - loss: 1.0597 - acc: 0.6433 - val_loss: 2.6736 - val_acc: 0.3546 Epoch 17/40 5200/5200 [==============================] - 153s 29ms/step - loss: 0.9034 - acc: 0.6987 - val_loss: 3.0697 - val_acc: 0.3585 Epoch 18/40 5200/5200 [==============================] - 145s 28ms/step - loss: 0.9025 - acc: 0.7046 - val_loss: 3.2138 - val_acc: 0.3492 Epoch 19/40 5200/5200 [==============================] - 147s 28ms/step - loss: 0.8246 - acc: 0.7233 - val_loss: 3.1570 - val_acc: 0.3715 Epoch 20/40 5200/5200 [==============================] - 152s 29ms/step - loss: 0.6798 - acc: 0.7765 - val_loss: 3.6677 - val_acc: 0.3300 Epoch 21/40 1696/5200 [========>.....................] - ETA: 1:38 - loss: 0.7017 - acc: 0.7718
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-2187-7934969a9897> in <module>() 40 second_iteration.compile(opt, scaled_loss, ['accuracy']) 41 ---> 42 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False) 43 44 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
'''class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 12479)
plt.imshow(shuffled_data[i][0])
l = second_iteration.predict(shuffled_data[i:1 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')'''
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 1299)
plt.imshow(shuffled_data[5200 + i][0])
l = second_iteration.predict(shuffled_data[5200 + i:5201 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[5200 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
if ind[0] != shuffled_y[5200 + i]: print(f'Confidence in the correct answer: {l[0][shuffled_y[5200 + i]]}')
print(f'Index: {i}')
Expected: flowers Got: fruit and vegetables Confidence: 0.7707887291908264 Confidence in the correct answer: 0.04208198934793472 Index: 450
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow import keras
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((1, 3), (3, 1), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.002)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.002)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.002)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.002)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.002)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.002)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=150, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dropout(0.15))
second_iteration.add(layers.Dense(units=80, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dropout(0.15))
second_iteration.add(layers.Dense(units=30, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 5200 samples, validate on 1300 samples Epoch 1/40 5200/5200 [==============================] - 104s 20ms/step - loss: 3.5529 - acc: 0.1163 - val_loss: 2.7514 - val_acc: 0.1631 Epoch 2/40 5200/5200 [==============================] - 99s 19ms/step - loss: 2.6529 - acc: 0.1644 - val_loss: 2.4834 - val_acc: 0.2008 Epoch 3/40 5200/5200 [==============================] - 103s 20ms/step - loss: 2.4951 - acc: 0.1967 - val_loss: 2.3792 - val_acc: 0.2031 Epoch 4/40 5200/5200 [==============================] - 103s 20ms/step - loss: 2.3899 - acc: 0.2235 - val_loss: 2.2912 - val_acc: 0.2308 Epoch 5/40 5200/5200 [==============================] - 101s 19ms/step - loss: 2.3394 - acc: 0.2390 - val_loss: 2.2791 - val_acc: 0.2415 Epoch 6/40 5200/5200 [==============================] - 105s 20ms/step - loss: 2.2965 - acc: 0.2498 - val_loss: 2.2205 - val_acc: 0.2808 Epoch 7/40 5200/5200 [==============================] - 105s 20ms/step - loss: 2.2474 - acc: 0.2788 - val_loss: 2.2183 - val_acc: 0.2846 Epoch 8/40 5200/5200 [==============================] - 99s 19ms/step - loss: 2.2097 - acc: 0.2915 - val_loss: 2.2179 - val_acc: 0.2831 Epoch 9/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.1771 - acc: 0.3073 - val_loss: 2.2092 - val_acc: 0.2769 Epoch 10/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.1297 - acc: 0.3175 - val_loss: 2.1656 - val_acc: 0.3062 Epoch 11/40 5200/5200 [==============================] - 100s 19ms/step - loss: 2.0894 - acc: 0.3373 - val_loss: 2.1361 - val_acc: 0.3246 Epoch 12/40 5200/5200 [==============================] - 103s 20ms/step - loss: 2.0344 - acc: 0.3625 - val_loss: 2.1033 - val_acc: 0.3546 Epoch 13/40 5200/5200 [==============================] - 102s 20ms/step - loss: 1.9998 - acc: 0.3756 - val_loss: 2.0658 - val_acc: 0.3792 Epoch 14/40 5200/5200 [==============================] - 97s 19ms/step - loss: 1.9523 - acc: 0.3960 - val_loss: 2.2458 - val_acc: 0.3485 Epoch 15/40 5200/5200 [==============================] - 97s 19ms/step - loss: 1.9123 - acc: 0.4162 - val_loss: 2.0980 - val_acc: 0.3900 Epoch 16/40 5200/5200 [==============================] - 97s 19ms/step - loss: 1.8913 - acc: 0.4369 - val_loss: 2.1176 - val_acc: 0.3900 Epoch 17/40 5200/5200 [==============================] - 102s 20ms/step - loss: 1.8682 - acc: 0.4408 - val_loss: 2.0802 - val_acc: 0.4054 Epoch 18/40 5200/5200 [==============================] - 98s 19ms/step - loss: 1.8068 - acc: 0.4617 - val_loss: 2.2312 - val_acc: 0.3815 Epoch 19/40 5200/5200 [==============================] - 97s 19ms/step - loss: 1.7819 - acc: 0.4798 - val_loss: 2.2010 - val_acc: 0.3854 Epoch 20/40 5200/5200 [==============================] - 97s 19ms/step - loss: 1.7415 - acc: 0.4971 - val_loss: 2.3168 - val_acc: 0.3746 Epoch 21/40 5200/5200 [==============================] - 97s 19ms/step - loss: 1.6822 - acc: 0.5188 - val_loss: 2.4357 - val_acc: 0.3662 Epoch 22/40 5200/5200 [==============================] - 97s 19ms/step - loss: 1.6318 - acc: 0.5352 - val_loss: 2.3850 - val_acc: 0.3838 Epoch 23/40 992/5200 [====>.........................] - ETA: 1:14 - loss: 1.5465 - acc: 0.5675
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-18-787620def109> in <module>() 46 second_iteration.compile(opt, scaled_loss, ['accuracy']) 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow import keras
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((1, 3), (3, 1), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=150, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=80, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=30, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 5200 samples, validate on 1300 samples Epoch 1/40 5200/5200 [==============================] - 98s 19ms/step - loss: 3.9937 - acc: 0.1102 - val_loss: 2.9581 - val_acc: 0.1377 Epoch 2/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.7676 - acc: 0.1594 - val_loss: 2.5657 - val_acc: 0.1708 Epoch 3/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.5469 - acc: 0.1819 - val_loss: 2.4887 - val_acc: 0.1992 Epoch 4/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.4370 - acc: 0.1998 - val_loss: 2.3585 - val_acc: 0.2146 Epoch 5/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.3562 - acc: 0.2227 - val_loss: 2.2944 - val_acc: 0.2215 Epoch 6/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.3080 - acc: 0.2412 - val_loss: 2.2739 - val_acc: 0.2385 Epoch 7/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.2754 - acc: 0.2488 - val_loss: 2.2436 - val_acc: 0.2477 Epoch 8/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.2372 - acc: 0.2750 - val_loss: 2.2721 - val_acc: 0.2615 Epoch 9/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.2247 - acc: 0.2783 - val_loss: 2.1795 - val_acc: 0.2800 Epoch 10/40 5200/5200 [==============================] - 98s 19ms/step - loss: 2.1844 - acc: 0.2965 - val_loss: 2.2228 - val_acc: 0.2962 Epoch 11/40 5200/5200 [==============================] - 98s 19ms/step - loss: 2.1602 - acc: 0.3065 - val_loss: 2.1910 - val_acc: 0.3115 Epoch 12/40 5200/5200 [==============================] - 96s 19ms/step - loss: 2.1479 - acc: 0.3142 - val_loss: 2.1176 - val_acc: 0.3246 Epoch 13/40 5200/5200 [==============================] - 97s 19ms/step - loss: 2.0977 - acc: 0.3365 - val_loss: 2.1536 - val_acc: 0.3377 Epoch 14/40 5200/5200 [==============================] - 100s 19ms/step - loss: 2.0756 - acc: 0.3500 - val_loss: 2.1457 - val_acc: 0.3523 Epoch 15/40 5200/5200 [==============================] - 101s 19ms/step - loss: 2.0553 - acc: 0.3485 - val_loss: 2.1781 - val_acc: 0.3485 Epoch 16/40 5200/5200 [==============================] - 101s 20ms/step - loss: 2.0192 - acc: 0.3687 - val_loss: 2.1935 - val_acc: 0.3431 Epoch 17/40 5200/5200 [==============================] - 104s 20ms/step - loss: 1.9903 - acc: 0.3790 - val_loss: 2.2019 - val_acc: 0.3562 Epoch 18/40 5200/5200 [==============================] - 103s 20ms/step - loss: 1.9751 - acc: 0.3950 - val_loss: 2.1948 - val_acc: 0.3554 Epoch 19/40 5200/5200 [==============================] - 100s 19ms/step - loss: 1.9549 - acc: 0.4069 - val_loss: 2.3154 - val_acc: 0.3423 Epoch 20/40 5200/5200 [==============================] - 103s 20ms/step - loss: 1.9216 - acc: 0.4163 - val_loss: 2.4805 - val_acc: 0.3669 Epoch 21/40 3648/5200 [====================>.........] - ETA: 28s - loss: 1.8961 - acc: 0.4268
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-19-c2e0792877a3> in <module>() 46 second_iteration.compile(opt, scaled_loss, ['accuracy']) 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
import tensorflow as tf
from tensorflow.keras import layers, models
from tensorflow import keras
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((1, 3), (3, 1), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
if True:
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(0.003)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=100, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=25, input_dim=50, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
second_iteration.add(layers.Dense(units=13, input_dim=25, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.001)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'Batch size 32:')
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Train on 5200 samples, validate on 1300 samples Epoch 1/40 5200/5200 [==============================] - 51s 10ms/step - loss: 2.8135 - acc: 0.1200 - val_loss: 2.5381 - val_acc: 0.1238 Epoch 2/40 5200/5200 [==============================] - 51s 10ms/step - loss: 2.4982 - acc: 0.1310 - val_loss: 2.5069 - val_acc: 0.1385 Epoch 3/40 5200/5200 [==============================] - 48s 9ms/step - loss: 2.4580 - acc: 0.1450 - val_loss: 2.3706 - val_acc: 0.1877 Epoch 4/40 5200/5200 [==============================] - 52s 10ms/step - loss: 2.3873 - acc: 0.1844 - val_loss: 2.3476 - val_acc: 0.1754 Epoch 5/40 5200/5200 [==============================] - 53s 10ms/step - loss: 2.3416 - acc: 0.2052 - val_loss: 2.2790 - val_acc: 0.2215 Epoch 6/40 5200/5200 [==============================] - 50s 10ms/step - loss: 2.3247 - acc: 0.2190 - val_loss: 2.3058 - val_acc: 0.2269 Epoch 7/40 5200/5200 [==============================] - 46s 9ms/step - loss: 2.2821 - acc: 0.2348 - val_loss: 2.2765 - val_acc: 0.2254 Epoch 8/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.2627 - acc: 0.2354 - val_loss: 2.2308 - val_acc: 0.2369 Epoch 9/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.2524 - acc: 0.2492 - val_loss: 2.2219 - val_acc: 0.2415 Epoch 10/40 5200/5200 [==============================] - 44s 8ms/step - loss: 2.2313 - acc: 0.2542 - val_loss: 2.2089 - val_acc: 0.2500 Epoch 11/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.2056 - acc: 0.2608 - val_loss: 2.2067 - val_acc: 0.2485 Epoch 12/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.1989 - acc: 0.2694 - val_loss: 2.1979 - val_acc: 0.2600 Epoch 13/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.1859 - acc: 0.2715 - val_loss: 2.1636 - val_acc: 0.2815 Epoch 14/40 5200/5200 [==============================] - 44s 8ms/step - loss: 2.1655 - acc: 0.2762 - val_loss: 2.1587 - val_acc: 0.2738 Epoch 15/40 5200/5200 [==============================] - 44s 8ms/step - loss: 2.1590 - acc: 0.2908 - val_loss: 2.1494 - val_acc: 0.2877 Epoch 16/40 5200/5200 [==============================] - 44s 8ms/step - loss: 2.1424 - acc: 0.2937 - val_loss: 2.1539 - val_acc: 0.2946 Epoch 17/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.1230 - acc: 0.3079 - val_loss: 2.1380 - val_acc: 0.2900 Epoch 18/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.1122 - acc: 0.3152 - val_loss: 2.1966 - val_acc: 0.2738 Epoch 19/40 5200/5200 [==============================] - 43s 8ms/step - loss: 2.0956 - acc: 0.3250 - val_loss: 2.1190 - val_acc: 0.3115 Epoch 20/40 5200/5200 [==============================] - 49s 10ms/step - loss: 2.0781 - acc: 0.3288 - val_loss: 2.1292 - val_acc: 0.3115 Epoch 21/40 5200/5200 [==============================] - 51s 10ms/step - loss: 2.0643 - acc: 0.3437 - val_loss: 2.1612 - val_acc: 0.2969 Epoch 22/40 5200/5200 [==============================] - 44s 8ms/step - loss: 2.0332 - acc: 0.3431 - val_loss: 2.1337 - val_acc: 0.2969 Epoch 23/40 5200/5200 [==============================] - 44s 8ms/step - loss: 2.0134 - acc: 0.3504 - val_loss: 2.1595 - val_acc: 0.3008 Epoch 24/40 5200/5200 [==============================] - 44s 8ms/step - loss: 2.0018 - acc: 0.3587 - val_loss: 2.1395 - val_acc: 0.3154 Epoch 25/40 5200/5200 [==============================] - 44s 8ms/step - loss: 1.9931 - acc: 0.3604 - val_loss: 2.1881 - val_acc: 0.3123 Epoch 26/40 5200/5200 [==============================] - 45s 9ms/step - loss: 1.9771 - acc: 0.3635 - val_loss: 2.1776 - val_acc: 0.3138 Epoch 27/40 1472/5200 [=======>......................] - ETA: 29s - loss: 1.9523 - acc: 0.3757
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-21-d3780413b670> in <module>() 46 second_iteration.compile(opt, scaled_loss, ['accuracy']) 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
In effect, the problem is reduced to solving the overfitting issue. I already have a model that can reach 77% accuracy on the training set in 20 epochs, now we just need to make it learn on validation as well.
Let's run another regularization grid search. 0.001, 0.004, 0.006, 0.008, 0.01.
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
for val in [0.001, 0.004, 0.006, 0.008, 0.01]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=80, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=60, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=13, input_dim=60, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=20, validation_split=0.2, shuffle=False, verbose=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.001 regularization 61.300000000000004% (train) vs. 35.300000000000004% (validation) Going 0.004 regularization 45.2% (train) vs. 37.9% (validation) Going 0.006 regularization 34.9% (train) vs. 30.5% (validation) Going 0.008 regularization
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-29-1e4f8c53cabb> in <module>() 41 second_iteration.compile(opt, scaled_loss, ['accuracy']) 42 ---> 43 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=20, validation_split=0.2, shuffle=False, verbose=False) 44 45 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
for val in [0.001]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 10)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 10)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 10)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 10)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 10)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 10)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=300, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=150, input_dim=300, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=75, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.001 regularization Train on 5200 samples, validate on 1300 samples Epoch 1/60 5200/5200 [==============================] - 150s 29ms/step - loss: 7.7465 - acc: 0.0869 - val_loss: 4.5929 - val_acc: 0.1277 Epoch 2/60 5200/5200 [==============================] - 147s 28ms/step - loss: 4.0051 - acc: 0.1173 - val_loss: 3.5111 - val_acc: 0.1262 Epoch 3/60 5200/5200 [==============================] - 144s 28ms/step - loss: 3.2742 - acc: 0.1385 - val_loss: 3.0110 - val_acc: 0.1438 Epoch 4/60 5200/5200 [==============================] - 144s 28ms/step - loss: 2.9146 - acc: 0.1540 - val_loss: 2.7711 - val_acc: 0.1785 Epoch 5/60 5200/5200 [==============================] - 143s 28ms/step - loss: 2.6929 - acc: 0.1865 - val_loss: 2.5704 - val_acc: 0.1885 Epoch 6/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.5715 - acc: 0.2012 - val_loss: 2.4452 - val_acc: 0.2208 Epoch 7/60 5200/5200 [==============================] - 143s 28ms/step - loss: 2.4838 - acc: 0.2165 - val_loss: 2.4069 - val_acc: 0.2392 Epoch 8/60 5200/5200 [==============================] - 146s 28ms/step - loss: 2.4387 - acc: 0.2206 - val_loss: 2.3577 - val_acc: 0.2123 Epoch 9/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.4203 - acc: 0.2175 - val_loss: 2.3136 - val_acc: 0.2377 Epoch 10/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.3632 - acc: 0.2356 - val_loss: 2.2965 - val_acc: 0.2462 Epoch 11/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.3277 - acc: 0.2448 - val_loss: 2.3004 - val_acc: 0.2223 Epoch 12/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.3057 - acc: 0.2469 - val_loss: 2.2421 - val_acc: 0.2531 Epoch 13/60 5200/5200 [==============================] - 143s 27ms/step - loss: 2.2726 - acc: 0.2577 - val_loss: 2.2343 - val_acc: 0.2585 Epoch 14/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.2815 - acc: 0.2571 - val_loss: 2.2310 - val_acc: 0.2638 Epoch 15/60 5200/5200 [==============================] - 143s 27ms/step - loss: 2.2515 - acc: 0.2687 - val_loss: 2.2400 - val_acc: 0.2754 Epoch 16/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.2623 - acc: 0.2623 - val_loss: 2.2250 - val_acc: 0.2762 Epoch 17/60 5200/5200 [==============================] - 142s 27ms/step - loss: 2.2459 - acc: 0.2658 - val_loss: 2.2189 - val_acc: 0.2746 Epoch 18/60 5200/5200 [==============================] - 141s 27ms/step - loss: 2.2411 - acc: 0.2737 - val_loss: 2.2133 - val_acc: 0.2662 Epoch 19/60 5200/5200 [==============================] - 144s 28ms/step - loss: 2.2439 - acc: 0.2737 - val_loss: 2.2105 - val_acc: 0.2877 Epoch 20/60 5200/5200 [==============================] - 144s 28ms/step - loss: 2.2213 - acc: 0.2790 - val_loss: 2.2051 - val_acc: 0.2700 Epoch 21/60 5200/5200 [==============================] - 144s 28ms/step - loss: 2.2269 - acc: 0.2823 - val_loss: 2.2131 - val_acc: 0.2746 Epoch 22/60 5200/5200 [==============================] - 145s 28ms/step - loss: 2.2184 - acc: 0.2873 - val_loss: 2.2109 - val_acc: 0.2785 Epoch 23/60 224/5200 [>.............................] - ETA: 2:11 - loss: 2.1425 - acc: 0.2768
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-40-22e60f0d993f> in <module>() 41 second_iteration.compile(opt, scaled_loss, ['accuracy']) 42 ---> 43 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False) 44 45 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
UPD: I found out that L1 regularization may be better for this project.
https://neptune.ai/blog/fighting-overfitting-with-l1-or-l2-regularization
L1 is "more robust to outliers", which through my "random EDA" (which consists of taking random validation images and looking at the model's thought process on them) seem to be very common. Rockets, for instance, could be considered outliers within the set of vehicles, because they don't resemble other vehicles such as cars and trains. It makes sense from a mathematical point of view that L1 regularization would work better in outlier-heavy datasets, because L2 regularization "inflates" the outlier weights through the squaring. You know what, let's try a bit of L1 regularization in this project. It could be that the model's failure to learn is due to our use of L2 regularization because it focuses on outliers, and in particular those in the training set. It also could be why overfitting is still witnessed in spite of regularization.
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
for val in [0.00025]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=300, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=150, input_dim=300, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=75, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.00025 regularization Train on 5200 samples, validate on 1300 samples Epoch 1/60 5200/5200 [==============================] - 176s 34ms/step - loss: 7.0836 - acc: 0.0948 - val_loss: 5.1205 - val_acc: 0.1354 Epoch 2/60 5200/5200 [==============================] - 166s 32ms/step - loss: 4.5498 - acc: 0.1313 - val_loss: 4.0118 - val_acc: 0.1508 Epoch 3/60 5200/5200 [==============================] - 166s 32ms/step - loss: 3.7711 - acc: 0.1619 - val_loss: 3.4573 - val_acc: 0.1800 Epoch 4/60 5200/5200 [==============================] - 166s 32ms/step - loss: 3.3334 - acc: 0.1854 - val_loss: 3.2223 - val_acc: 0.1692 Epoch 5/60 5200/5200 [==============================] - 166s 32ms/step - loss: 3.0562 - acc: 0.1963 - val_loss: 2.8991 - val_acc: 0.2146 Epoch 6/60 5200/5200 [==============================] - 166s 32ms/step - loss: 2.8602 - acc: 0.2088 - val_loss: 2.7531 - val_acc: 0.2323 Epoch 7/60 5200/5200 [==============================] - 166s 32ms/step - loss: 2.7243 - acc: 0.2308 - val_loss: 2.6507 - val_acc: 0.2485 Epoch 8/60 5200/5200 [==============================] - 166s 32ms/step - loss: 2.6260 - acc: 0.2352 - val_loss: 2.5331 - val_acc: 0.2408 Epoch 9/60 5200/5200 [==============================] - 165s 32ms/step - loss: 2.5511 - acc: 0.2406 - val_loss: 2.4872 - val_acc: 0.2408 Epoch 10/60 5200/5200 [==============================] - 3706s 713ms/step - loss: 2.5035 - acc: 0.2521 - val_loss: 2.4687 - val_acc: 0.2454 Epoch 11/60 5200/5200 [==============================] - 113s 22ms/step - loss: 2.4705 - acc: 0.2481 - val_loss: 2.4337 - val_acc: 0.2446 Epoch 12/60 5200/5200 [==============================] - 140s 27ms/step - loss: 2.4362 - acc: 0.2554 - val_loss: 2.3970 - val_acc: 0.2569 Epoch 13/60 5200/5200 [==============================] - 147s 28ms/step - loss: 2.4087 - acc: 0.2535 - val_loss: 2.3898 - val_acc: 0.2585 Epoch 14/60 5200/5200 [==============================] - 152s 29ms/step - loss: 2.3942 - acc: 0.2621 - val_loss: 2.3819 - val_acc: 0.2608 Epoch 15/60 5200/5200 [==============================] - 144s 28ms/step - loss: 2.3611 - acc: 0.2688 - val_loss: 2.3503 - val_acc: 0.2677 Epoch 16/60 5200/5200 [==============================] - 143s 28ms/step - loss: 2.3574 - acc: 0.2706 - val_loss: 2.3553 - val_acc: 0.2646 Epoch 17/60 5200/5200 [==============================] - 145s 28ms/step - loss: 2.3401 - acc: 0.2713 - val_loss: 2.3830 - val_acc: 0.2700 Epoch 18/60 5200/5200 [==============================] - 143s 28ms/step - loss: 2.3129 - acc: 0.2773 - val_loss: 2.3428 - val_acc: 0.2638 Epoch 19/60 5200/5200 [==============================] - 144s 28ms/step - loss: 2.2993 - acc: 0.2852 - val_loss: 2.3439 - val_acc: 0.2700 Epoch 20/60 5200/5200 [==============================] - 144s 28ms/step - loss: 2.2923 - acc: 0.2915 - val_loss: 2.3552 - val_acc: 0.2631 Epoch 21/60 5200/5200 [==============================] - 143s 27ms/step - loss: 2.2930 - acc: 0.2846 - val_loss: 2.3160 - val_acc: 0.2700 Epoch 22/60 5200/5200 [==============================] - 131s 25ms/step - loss: 2.2919 - acc: 0.2933 - val_loss: 2.3013 - val_acc: 0.2746 Epoch 23/60 5200/5200 [==============================] - 95s 18ms/step - loss: 2.2744 - acc: 0.2937 - val_loss: 2.3313 - val_acc: 0.2908 Epoch 24/60 5200/5200 [==============================] - 95s 18ms/step - loss: 2.2568 - acc: 0.3008 - val_loss: 2.2964 - val_acc: 0.2938 Epoch 25/60 5200/5200 [==============================] - 93s 18ms/step - loss: 2.2721 - acc: 0.3021 - val_loss: 2.3121 - val_acc: 0.2846 Epoch 26/60 5200/5200 [==============================] - 93s 18ms/step - loss: 2.2485 - acc: 0.3167 - val_loss: 2.2923 - val_acc: 0.2977 Epoch 27/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.2369 - acc: 0.3246 - val_loss: 2.2656 - val_acc: 0.3085 Epoch 28/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.2224 - acc: 0.3298 - val_loss: 2.2955 - val_acc: 0.3092 Epoch 29/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.2072 - acc: 0.3312 - val_loss: 2.2505 - val_acc: 0.3077 Epoch 30/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.2093 - acc: 0.3363 - val_loss: 2.2365 - val_acc: 0.3331 Epoch 31/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.2034 - acc: 0.3433 - val_loss: 2.2431 - val_acc: 0.3315 Epoch 32/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1926 - acc: 0.3387 - val_loss: 2.2579 - val_acc: 0.3246 Epoch 33/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1772 - acc: 0.3469 - val_loss: 2.2768 - val_acc: 0.3231 Epoch 34/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1681 - acc: 0.3558 - val_loss: 2.2301 - val_acc: 0.3246 Epoch 35/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1595 - acc: 0.3619 - val_loss: 2.2836 - val_acc: 0.3223 Epoch 36/60 5200/5200 [==============================] - 95s 18ms/step - loss: 2.1842 - acc: 0.3579 - val_loss: 2.2973 - val_acc: 0.3254 Epoch 37/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1737 - acc: 0.3594 - val_loss: 2.2958 - val_acc: 0.3177 Epoch 38/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1650 - acc: 0.3606 - val_loss: 2.2874 - val_acc: 0.3338 Epoch 39/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1627 - acc: 0.3556 - val_loss: 2.2883 - val_acc: 0.3208 Epoch 40/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1435 - acc: 0.3669 - val_loss: 2.3293 - val_acc: 0.3138 Epoch 41/60 5200/5200 [==============================] - 92s 18ms/step - loss: 2.1480 - acc: 0.3644 - val_loss: 2.2911 - val_acc: 0.3223 Epoch 42/60 5200/5200 [==============================] - 91s 18ms/step - loss: 2.1358 - acc: 0.3665 - val_loss: 2.3137 - val_acc: 0.3369 Epoch 43/60 5200/5200 [==============================] - 98s 19ms/step - loss: 2.1470 - acc: 0.3662 - val_loss: 2.3292 - val_acc: 0.3346 Epoch 44/60 5200/5200 [==============================] - 97s 19ms/step - loss: 2.1171 - acc: 0.3862 - val_loss: 2.3341 - val_acc: 0.3277 Epoch 45/60 5200/5200 [==============================] - 109s 21ms/step - loss: 2.1203 - acc: 0.3773 - val_loss: 2.3394 - val_acc: 0.3469 Epoch 46/60 5200/5200 [==============================] - 108s 21ms/step - loss: 2.1321 - acc: 0.3856 - val_loss: 2.3678 - val_acc: 0.3362 Epoch 47/60 5200/5200 [==============================] - 107s 21ms/step - loss: 2.1522 - acc: 0.3815 - val_loss: 2.4460 - val_acc: 0.3262 Epoch 48/60 3616/5200 [===================>..........] - ETA: 28s - loss: 2.1042 - acc: 0.3946
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-44-2b8310666962> in <module>() 41 second_iteration.compile(opt, scaled_loss, ['accuracy']) 42 ---> 43 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False) 44 45 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
'''class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 12479)
plt.imshow(shuffled_data[i][0])
l = second_iteration.predict(shuffled_data[i:1 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')'''
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 1299)
plt.imshow(shuffled_data[5200 + i][0])
l = second_iteration.predict(shuffled_data[5200 + i:5201 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[5200 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
if ind[0] != shuffled_y[5200 + i]: print(f'Confidence in the correct answer: {l[0][shuffled_y[5200 + i]]}')
print(f'Index: {i}')
Expected: electronics Got: food containers Confidence: 0.2391107827425003 Confidence in the correct answer: 0.20318059623241425 Index: 1237
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
for val in [0.0004]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=100, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=300, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=150, input_dim=300, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=75, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[5200:], one_hot_encode(shuffled_y[5200:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0004 regularization Train on 5200 samples, validate on 1300 samples Epoch 1/60 5200/5200 [==============================] - 216s 42ms/step - loss: 10.0755 - acc: 0.1013 - val_loss: 6.6721 - val_acc: 0.1338 Epoch 2/60 5200/5200 [==============================] - 203s 39ms/step - loss: 5.7217 - acc: 0.1344 - val_loss: 4.9522 - val_acc: 0.1392 Epoch 3/60 5200/5200 [==============================] - 203s 39ms/step - loss: 4.4563 - acc: 0.1463 - val_loss: 4.0082 - val_acc: 0.1769 Epoch 4/60 5200/5200 [==============================] - 202s 39ms/step - loss: 3.7399 - acc: 0.1565 - val_loss: 3.4425 - val_acc: 0.1785 Epoch 5/60 5200/5200 [==============================] - 202s 39ms/step - loss: 3.3013 - acc: 0.1769 - val_loss: 3.1448 - val_acc: 0.1923 Epoch 6/60 5200/5200 [==============================] - 202s 39ms/step - loss: 3.0352 - acc: 0.1746 - val_loss: 2.8968 - val_acc: 0.2062 Epoch 7/60 5200/5200 [==============================] - 202s 39ms/step - loss: 2.8532 - acc: 0.1973 - val_loss: 2.8402 - val_acc: 0.2069 Epoch 8/60 5200/5200 [==============================] - 208s 40ms/step - loss: 2.7435 - acc: 0.1969 - val_loss: 2.6676 - val_acc: 0.2108 Epoch 9/60 480/5200 [=>............................] - ETA: 2:59 - loss: 2.6947 - acc: 0.1896
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-79-7edbf0b5eede> in <module>() 41 second_iteration.compile(opt, scaled_loss, ['accuracy']) 42 ---> 43 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False) 44 45 tr_acc = second_iteration.evaluate(shuffled_data[:5200], one_hot_encode(shuffled_y[:5200]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
Today is the last day of November. I performed some more EDA and decided that reptile images with blue background should be removed as they are virtually indistinguishable from sea animals. No, genuinely.
Also, the size of the dataset will be reduced once again to 2600+650, because then less complexity is needed which 1) reduces training time and 2) reduces overfitting.
Detecting a blue background in reptile images is kind of tough. I'll experiment with different methods.
condensed_data = np.zeros((6500, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((6500), dtype=np.uint8)
ptrs = [500] * 13
training = 0
validation = 0
bluerept = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if val == 10: continue
if ptrs[val] == 0: continue
blueback = False
for k in range(30):
for l in range(30):
cells = []
cells.append(classified_data_2[i][0][k][l])
cells.append(classified_data_2[i][0][k+1][l])
cells.append(classified_data_2[i][0][k+2][l])
cells.append(classified_data_2[i][0][k][l+1])
cells.append(classified_data_2[i][0][k+1][l+1])
cells.append(classified_data_2[i][0][k+2][l+1])
cells.append(classified_data_2[i][0][k][l+2])
cells.append(classified_data_2[i][0][k+1][l+2])
cells.append(classified_data_2[i][0][k+2][l+2])
cnt = 0
for cell in cells:
if (int(cell[2]) / 10 >= int(cell[0]) + int(cell[1])):
cnt += 1
if cnt == 9:
blueback = True
break
if blueback: break
if blueback: bluerept += 1
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: continue
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][0][k][l][0],classified_data_2[i][0][k][l][1],classified_data_2[i][0][k][l][2],\
classified_data_2[i][0][k+1][l][0],classified_data_2[i][0][k+1][l][1],classified_data_2[i][0][k+1][l][2],\
classified_data_2[i][0][k][l+1][0],classified_data_2[i][0][k][l+1][1],classified_data_2[i][0][k][l+1][2],\
classified_data_2[i][0][k+1][l+1][0],classified_data_2[i][0][k+1][l+1][1],classified_data_2[i][0][k+1][l+1][2])==255:
found = True
break
if found: break
if found: continue
if (training + validation) % 100 == 0: print(training + validation, end=" ")
if ptrs[val] <= 100:
condensed_data[5200 + validation] = classified_data_2[i]
condensed_y[5200 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((6500, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((6500), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(5200)]
shuffled_valid = [*range(5200, 6500)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(5200):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(1300):
shuffled_data[shuffled_valid[i]] = condensed_data[5200 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[5200 + i]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900
bluerept
77
Alright, then.
condensed_data = np.zeros((3250, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((3250), dtype=np.uint8)
ptrs = [250] * 13
training = 0
validation = 0
bluerept = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
blueback = False
for k in range(30):
for l in range(30):
cells = []
cells.append(classified_data_2[i][0][k][l])
cells.append(classified_data_2[i][0][k+1][l])
cells.append(classified_data_2[i][0][k+2][l])
cells.append(classified_data_2[i][0][k][l+1])
cells.append(classified_data_2[i][0][k+1][l+1])
cells.append(classified_data_2[i][0][k+2][l+1])
cells.append(classified_data_2[i][0][k][l+2])
cells.append(classified_data_2[i][0][k+1][l+2])
cells.append(classified_data_2[i][0][k+2][l+2])
cnt = 0
for cell in cells:
if (int(cell[2]) / 10 >= int(cell[0]) + int(cell[1])):
cnt += 1
if cnt == 9:
blueback = True
break
if blueback: break
if blueback and val == 10: continue
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: continue
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][0][k][l][0],classified_data_2[i][0][k][l][1],classified_data_2[i][0][k][l][2],\
classified_data_2[i][0][k+1][l][0],classified_data_2[i][0][k+1][l][1],classified_data_2[i][0][k+1][l][2],\
classified_data_2[i][0][k][l+1][0],classified_data_2[i][0][k][l+1][1],classified_data_2[i][0][k][l+1][2],\
classified_data_2[i][0][k+1][l+1][0],classified_data_2[i][0][k+1][l+1][1],classified_data_2[i][0][k+1][l+1][2])==255:
found = True
break
if found: break
if found: continue
if (training + validation) % 100 == 0: print(training + validation, end=" ")
if ptrs[val] <= 50:
condensed_data[2600 + validation] = classified_data_2[i]
condensed_y[2600 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((3250, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((3250), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(2600)]
shuffled_valid = [*range(2600, 3250)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(2600):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(650):
shuffled_data[shuffled_valid[i]] = condensed_data[2600 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[2600 + i]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
for val in [0.00025]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=150, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dense(units=50, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.00025 regularization Train on 2600 samples, validate on 650 samples Epoch 1/60 2600/2600 [==============================] - 42s 16ms/step - loss: 3.7685 - acc: 0.1177 - val_loss: 3.0582 - val_acc: 0.1708 Epoch 2/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.9237 - acc: 0.1662 - val_loss: 2.8041 - val_acc: 0.1415 Epoch 3/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.7390 - acc: 0.1854 - val_loss: 2.6604 - val_acc: 0.1908 Epoch 4/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.6221 - acc: 0.1996 - val_loss: 2.5389 - val_acc: 0.2077 Epoch 5/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.5291 - acc: 0.2181 - val_loss: 2.4750 - val_acc: 0.2108 Epoch 6/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.4707 - acc: 0.2265 - val_loss: 2.4267 - val_acc: 0.2231 Epoch 7/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.4329 - acc: 0.2327 - val_loss: 2.4048 - val_acc: 0.2662 Epoch 8/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3896 - acc: 0.2519 - val_loss: 2.3942 - val_acc: 0.2462 Epoch 9/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3534 - acc: 0.2638 - val_loss: 2.4250 - val_acc: 0.2569 Epoch 10/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3242 - acc: 0.2635 - val_loss: 2.3683 - val_acc: 0.2615 Epoch 11/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3056 - acc: 0.2600 - val_loss: 2.3587 - val_acc: 0.2662 Epoch 12/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.2719 - acc: 0.2769 - val_loss: 2.4041 - val_acc: 0.2554 Epoch 13/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.2548 - acc: 0.2838 - val_loss: 2.3375 - val_acc: 0.2892 Epoch 14/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.2377 - acc: 0.2846 - val_loss: 2.3200 - val_acc: 0.2862 Epoch 15/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.2051 - acc: 0.3015 - val_loss: 2.3309 - val_acc: 0.2862 Epoch 16/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.1841 - acc: 0.3208 - val_loss: 2.3212 - val_acc: 0.3000 Epoch 17/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.1496 - acc: 0.3158 - val_loss: 2.3385 - val_acc: 0.3031 Epoch 18/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.1563 - acc: 0.3242 - val_loss: 2.2724 - val_acc: 0.3062 Epoch 19/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.1324 - acc: 0.3273 - val_loss: 2.4154 - val_acc: 0.2785 Epoch 20/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.0950 - acc: 0.3469 - val_loss: 2.2914 - val_acc: 0.3185 Epoch 21/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.0432 - acc: 0.3746 - val_loss: 2.3606 - val_acc: 0.2923 Epoch 22/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.0130 - acc: 0.3942 - val_loss: 2.3385 - val_acc: 0.3323 Epoch 23/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.9981 - acc: 0.3950 - val_loss: 2.4388 - val_acc: 0.2923 Epoch 24/60 2600/2600 [==============================] - 32s 12ms/step - loss: 1.9655 - acc: 0.4096 - val_loss: 2.3452 - val_acc: 0.3354 Epoch 25/60 2600/2600 [==============================] - 32s 12ms/step - loss: 1.9952 - acc: 0.3942 - val_loss: 2.3717 - val_acc: 0.3062 Epoch 26/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.9475 - acc: 0.4181 - val_loss: 2.3595 - val_acc: 0.3415 Epoch 27/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.8502 - acc: 0.4504 - val_loss: 2.3371 - val_acc: 0.3554 Epoch 28/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.8454 - acc: 0.4604 - val_loss: 2.5106 - val_acc: 0.3200 Epoch 29/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.8561 - acc: 0.4665 - val_loss: 2.4307 - val_acc: 0.3369 Epoch 30/60 2600/2600 [==============================] - 32s 12ms/step - loss: 1.8081 - acc: 0.4785 - val_loss: 2.5986 - val_acc: 0.3354 Epoch 31/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.7411 - acc: 0.5019 - val_loss: 2.6004 - val_acc: 0.3200 Epoch 32/60 2600/2600 [==============================] - 32s 12ms/step - loss: 1.8017 - acc: 0.4927 - val_loss: 2.6945 - val_acc: 0.3154 Epoch 33/60 2600/2600 [==============================] - 32s 12ms/step - loss: 1.8480 - acc: 0.4827 - val_loss: 2.7928 - val_acc: 0.2985 Epoch 34/60 2600/2600 [==============================] - 33s 13ms/step - loss: 1.8453 - acc: 0.4877 - val_loss: 2.6464 - val_acc: 0.3477 Epoch 35/60 2600/2600 [==============================] - 32s 12ms/step - loss: 1.6923 - acc: 0.5277 - val_loss: 2.7588 - val_acc: 0.3277 Epoch 36/60 2600/2600 [==============================] - 33s 13ms/step - loss: 1.6132 - acc: 0.5504 - val_loss: 2.7701 - val_acc: 0.3200 Epoch 37/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.5923 - acc: 0.5554 - val_loss: 2.9143 - val_acc: 0.3338 Epoch 38/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.5577 - acc: 0.5727 - val_loss: 2.9525 - val_acc: 0.3138 Epoch 39/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.5612 - acc: 0.5815 - val_loss: 3.0013 - val_acc: 0.3154 Epoch 40/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.4758 - acc: 0.5981 - val_loss: 3.0659 - val_acc: 0.3231 Epoch 41/60 2600/2600 [==============================] - 31s 12ms/step - loss: 1.4836 - acc: 0.5969 - val_loss: 3.0244 - val_acc: 0.3385 Epoch 42/60 1952/2600 [=====================>........] - ETA: 7s - loss: 1.4488 - acc: 0.6199
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-95-3d50a0d3cf3f> in <module>() 38 second_iteration.compile(opt, scaled_loss, ['accuracy']) 39 ---> 40 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False) 41 42 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.2)
return cce
for val in [0.0005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=50, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dense(units=30, input_dim=50, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0005 regularization Train on 2600 samples, validate on 650 samples Epoch 1/60 2600/2600 [==============================] - 43s 17ms/step - loss: 4.2231 - acc: 0.0912 - val_loss: 3.1436 - val_acc: 0.0892 Epoch 2/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.9090 - acc: 0.1158 - val_loss: 2.7126 - val_acc: 0.1662 Epoch 3/60 2600/2600 [==============================] - 33s 13ms/step - loss: 2.6756 - acc: 0.1519 - val_loss: 2.5832 - val_acc: 0.1677 Epoch 4/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.5613 - acc: 0.1723 - val_loss: 2.4827 - val_acc: 0.1862 Epoch 5/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4922 - acc: 0.1831 - val_loss: 2.4178 - val_acc: 0.2092 Epoch 6/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4361 - acc: 0.1962 - val_loss: 2.3955 - val_acc: 0.2169 Epoch 7/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.4156 - acc: 0.2081 - val_loss: 2.3798 - val_acc: 0.2154 Epoch 8/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3875 - acc: 0.2077 - val_loss: 2.3581 - val_acc: 0.2108 Epoch 9/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3650 - acc: 0.2127 - val_loss: 2.3410 - val_acc: 0.2354 Epoch 10/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3637 - acc: 0.2188 - val_loss: 2.3193 - val_acc: 0.2277 Epoch 11/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3421 - acc: 0.2238 - val_loss: 2.3264 - val_acc: 0.2369 Epoch 12/60 2600/2600 [==============================] - 33s 13ms/step - loss: 2.3288 - acc: 0.2323 - val_loss: 2.3051 - val_acc: 0.2492 Epoch 13/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3149 - acc: 0.2408 - val_loss: 2.3193 - val_acc: 0.2431 Epoch 14/60 2600/2600 [==============================] - 33s 13ms/step - loss: 2.3038 - acc: 0.2346 - val_loss: 2.3229 - val_acc: 0.2369 Epoch 15/60 2600/2600 [==============================] - 33s 13ms/step - loss: 2.2966 - acc: 0.2454 - val_loss: 2.3036 - val_acc: 0.2631 Epoch 16/60 1568/2600 [=================>............] - ETA: 12s - loss: 2.2648 - acc: 0.2532
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-104-acdca89ee6d1> in <module>() 38 second_iteration.compile(opt, scaled_loss, ['accuracy']) 39 ---> 40 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False) 41 42 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.3)
return cce
for val in [0.0005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=40, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=30, input_dim=40, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=30, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0005 regularization Train on 2600 samples, validate on 650 samples Epoch 1/60 2600/2600 [==============================] - 44s 17ms/step - loss: 4.0797 - acc: 0.0796 - val_loss: 3.0335 - val_acc: 0.0800 Epoch 2/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.8048 - acc: 0.0842 - val_loss: 2.6840 - val_acc: 0.1185 Epoch 3/60 2600/2600 [==============================] - 33s 13ms/step - loss: 2.6040 - acc: 0.1277 - val_loss: 2.5358 - val_acc: 0.1338 Epoch 4/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.5460 - acc: 0.1408 - val_loss: 2.5027 - val_acc: 0.1354 Epoch 5/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.5212 - acc: 0.1404 - val_loss: 2.4854 - val_acc: 0.1262 Epoch 6/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.5051 - acc: 0.1365 - val_loss: 2.4718 - val_acc: 0.1354 Epoch 7/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4922 - acc: 0.1335 - val_loss: 2.4590 - val_acc: 0.1400 Epoch 8/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4734 - acc: 0.1388 - val_loss: 2.4388 - val_acc: 0.1492 Epoch 9/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4603 - acc: 0.1354 - val_loss: 2.4318 - val_acc: 0.1385 Epoch 10/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4534 - acc: 0.1327 - val_loss: 2.4315 - val_acc: 0.1262 Epoch 11/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4484 - acc: 0.1365 - val_loss: 2.4158 - val_acc: 0.1662 Epoch 12/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.4409 - acc: 0.1438 - val_loss: 2.4109 - val_acc: 0.1338 Epoch 13/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4351 - acc: 0.1419 - val_loss: 2.4003 - val_acc: 0.1385 Epoch 14/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4270 - acc: 0.1577 - val_loss: 2.3914 - val_acc: 0.1400 Epoch 15/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4257 - acc: 0.1500 - val_loss: 2.3995 - val_acc: 0.1538 Epoch 16/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4253 - acc: 0.1369 - val_loss: 2.3903 - val_acc: 0.1462 Epoch 17/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4266 - acc: 0.1481 - val_loss: 2.3845 - val_acc: 0.1554 Epoch 18/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4119 - acc: 0.1569 - val_loss: 2.3796 - val_acc: 0.1538 Epoch 19/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4138 - acc: 0.1523 - val_loss: 2.3759 - val_acc: 0.1569 Epoch 20/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4051 - acc: 0.1635 - val_loss: 2.3767 - val_acc: 0.1462 Epoch 21/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4165 - acc: 0.1515 - val_loss: 2.3704 - val_acc: 0.1538 Epoch 22/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.4033 - acc: 0.1600 - val_loss: 2.3651 - val_acc: 0.1677 Epoch 23/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3955 - acc: 0.1669 - val_loss: 2.3636 - val_acc: 0.1815 Epoch 24/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3904 - acc: 0.1681 - val_loss: 2.3610 - val_acc: 0.1662 Epoch 25/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3925 - acc: 0.1688 - val_loss: 2.3566 - val_acc: 0.1615 Epoch 26/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3840 - acc: 0.1731 - val_loss: 2.3472 - val_acc: 0.1692 Epoch 27/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3822 - acc: 0.1723 - val_loss: 2.3430 - val_acc: 0.1677 Epoch 28/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3896 - acc: 0.1792 - val_loss: 2.3422 - val_acc: 0.1631 Epoch 29/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3738 - acc: 0.1827 - val_loss: 2.3385 - val_acc: 0.1662 Epoch 30/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3811 - acc: 0.1754 - val_loss: 2.3376 - val_acc: 0.1569 Epoch 31/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3701 - acc: 0.1796 - val_loss: 2.3376 - val_acc: 0.1600 Epoch 32/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3698 - acc: 0.1831 - val_loss: 2.3334 - val_acc: 0.1585 Epoch 33/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3702 - acc: 0.1865 - val_loss: 2.3348 - val_acc: 0.1677 Epoch 34/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3653 - acc: 0.1873 - val_loss: 2.3405 - val_acc: 0.2092 Epoch 35/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3662 - acc: 0.1823 - val_loss: 2.3252 - val_acc: 0.1708 Epoch 36/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3601 - acc: 0.1896 - val_loss: 2.3314 - val_acc: 0.1692 Epoch 37/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3623 - acc: 0.1977 - val_loss: 2.3359 - val_acc: 0.2015 Epoch 38/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3548 - acc: 0.1846 - val_loss: 2.3278 - val_acc: 0.1831 Epoch 39/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3624 - acc: 0.1912 - val_loss: 2.3681 - val_acc: 0.1708 Epoch 40/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3608 - acc: 0.1942 - val_loss: 2.3267 - val_acc: 0.1923 Epoch 41/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3548 - acc: 0.1865 - val_loss: 2.3279 - val_acc: 0.1708 Epoch 42/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3387 - acc: 0.1969 - val_loss: 2.3349 - val_acc: 0.1785 Epoch 43/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3650 - acc: 0.1892 - val_loss: 2.3304 - val_acc: 0.1831 Epoch 44/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3546 - acc: 0.2062 - val_loss: 2.3271 - val_acc: 0.1877 Epoch 45/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3477 - acc: 0.2062 - val_loss: 2.3284 - val_acc: 0.1908 Epoch 46/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3388 - acc: 0.2100 - val_loss: 2.3182 - val_acc: 0.1846 Epoch 47/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3406 - acc: 0.1942 - val_loss: 2.3228 - val_acc: 0.1923 Epoch 48/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3368 - acc: 0.2004 - val_loss: 2.3169 - val_acc: 0.1985 Epoch 49/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3469 - acc: 0.1946 - val_loss: 2.3069 - val_acc: 0.2000 Epoch 50/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3364 - acc: 0.2085 - val_loss: 2.3175 - val_acc: 0.1862 Epoch 51/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3281 - acc: 0.2173 - val_loss: 2.3075 - val_acc: 0.2062 Epoch 52/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3296 - acc: 0.2081 - val_loss: 2.3225 - val_acc: 0.2046 Epoch 53/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3373 - acc: 0.2112 - val_loss: 2.3130 - val_acc: 0.1938 Epoch 54/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3255 - acc: 0.2146 - val_loss: 2.3223 - val_acc: 0.1985 Epoch 55/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3288 - acc: 0.2104 - val_loss: 2.3065 - val_acc: 0.2031 Epoch 56/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3298 - acc: 0.2085 - val_loss: 2.3034 - val_acc: 0.2031 Epoch 57/60 2600/2600 [==============================] - 31s 12ms/step - loss: 2.3270 - acc: 0.2050 - val_loss: 2.3112 - val_acc: 0.2031 Epoch 58/60 2600/2600 [==============================] - 33s 13ms/step - loss: 2.3193 - acc: 0.2169 - val_loss: 2.3203 - val_acc: 0.2215 Epoch 59/60 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3185 - acc: 0.2092 - val_loss: 2.3180 - val_acc: 0.2046 Epoch 60/60 2600/2600 [==============================] - 33s 13ms/step - loss: 2.3260 - acc: 0.2138 - val_loss: 2.3164 - val_acc: 0.2108 22.700000000000003% (train) vs. 21.0% (validation)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
for val in [0.0003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=100, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=100, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0003 regularization Train on 2600 samples, validate on 650 samples Epoch 1/100 2600/2600 [==============================] - 49s 19ms/step - loss: 4.1914 - acc: 0.1112 - val_loss: 3.2640 - val_acc: 0.0908 Epoch 2/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.9859 - acc: 0.1569 - val_loss: 2.7907 - val_acc: 0.1862 Epoch 3/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.7415 - acc: 0.1704 - val_loss: 2.7033 - val_acc: 0.1938 Epoch 4/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.6375 - acc: 0.1938 - val_loss: 2.5614 - val_acc: 0.1985 Epoch 5/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.5576 - acc: 0.2046 - val_loss: 2.4870 - val_acc: 0.2185 Epoch 6/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.5174 - acc: 0.2027 - val_loss: 2.4878 - val_acc: 0.2062 Epoch 7/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.4757 - acc: 0.2242 - val_loss: 2.4263 - val_acc: 0.2446 Epoch 8/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.4645 - acc: 0.2204 - val_loss: 2.4270 - val_acc: 0.2277 Epoch 9/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.4252 - acc: 0.2254 - val_loss: 2.3991 - val_acc: 0.2400 Epoch 10/100 2600/2600 [==============================] - 34s 13ms/step - loss: 2.3821 - acc: 0.2427 - val_loss: 2.3808 - val_acc: 0.2492 Epoch 11/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.3643 - acc: 0.2554 - val_loss: 2.3618 - val_acc: 0.2646 Epoch 12/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.3340 - acc: 0.2696 - val_loss: 2.3507 - val_acc: 0.2677 Epoch 13/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.3165 - acc: 0.2673 - val_loss: 2.3535 - val_acc: 0.2738 Epoch 14/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.2914 - acc: 0.2781 - val_loss: 2.3493 - val_acc: 0.2738 Epoch 15/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.2819 - acc: 0.2735 - val_loss: 2.3600 - val_acc: 0.2723 Epoch 16/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.2677 - acc: 0.2835 - val_loss: 2.3420 - val_acc: 0.2723 Epoch 17/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.2431 - acc: 0.2973 - val_loss: 2.3654 - val_acc: 0.2846 Epoch 18/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.2392 - acc: 0.2938 - val_loss: 2.3652 - val_acc: 0.2908 Epoch 19/100 2600/2600 [==============================] - 34s 13ms/step - loss: 2.2081 - acc: 0.3112 - val_loss: 2.3627 - val_acc: 0.2923 Epoch 20/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.1928 - acc: 0.3077 - val_loss: 2.3636 - val_acc: 0.2769 Epoch 21/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.1694 - acc: 0.3131 - val_loss: 2.3919 - val_acc: 0.2815 Epoch 22/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.1758 - acc: 0.3162 - val_loss: 2.5021 - val_acc: 0.2785 Epoch 23/100 2600/2600 [==============================] - 32s 12ms/step - loss: 2.1512 - acc: 0.3438 - val_loss: 2.4762 - val_acc: 0.2662 Epoch 24/100 2600/2600 [==============================] - 33s 13ms/step - loss: 2.1489 - acc: 0.3327 - val_loss: 2.4640 - val_acc: 0.2754 Epoch 25/100 1504/2600 [================>.............] - ETA: 12s - loss: 2.1187 - acc: 0.3511
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-115-50a0f1db0ae6> in <module>() 39 second_iteration.compile(opt, scaled_loss, ['accuracy']) 40 ---> 41 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=100, validation_split=0.2, shuffle=False) 42 43 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
for val in [0.0007]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=80, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=80, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val * 1.5)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=100, validation_split=0.2, shuffle=False)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'{tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0007 regularization Train on 2600 samples, validate on 650 samples Epoch 1/100 2600/2600 [==============================] - 91s 35ms/step - loss: 20.7730 - acc: 0.1212 - val_loss: 11.5462 - val_acc: 0.1738 Epoch 2/100 2600/2600 [==============================] - 75s 29ms/step - loss: 9.0324 - acc: 0.1773 - val_loss: 7.3397 - val_acc: 0.1754 Epoch 3/100 2600/2600 [==============================] - 75s 29ms/step - loss: 6.5371 - acc: 0.1885 - val_loss: 5.8816 - val_acc: 0.1554 Epoch 4/100 2600/2600 [==============================] - 77s 30ms/step - loss: 5.3358 - acc: 0.1900 - val_loss: 4.8231 - val_acc: 0.2123 Epoch 5/100 2600/2600 [==============================] - 76s 29ms/step - loss: 4.5456 - acc: 0.2004 - val_loss: 4.2077 - val_acc: 0.2262 Epoch 6/100 2600/2600 [==============================] - 75s 29ms/step - loss: 4.0408 - acc: 0.2142 - val_loss: 3.7783 - val_acc: 0.2185 Epoch 7/100 2600/2600 [==============================] - 76s 29ms/step - loss: 3.6667 - acc: 0.2173 - val_loss: 3.5152 - val_acc: 0.2138 Epoch 8/100 2600/2600 [==============================] - 75s 29ms/step - loss: 3.4256 - acc: 0.2308 - val_loss: 3.2807 - val_acc: 0.2015 Epoch 9/100 2600/2600 [==============================] - 76s 29ms/step - loss: 3.2064 - acc: 0.2227 - val_loss: 3.1323 - val_acc: 0.2123 Epoch 10/100 2600/2600 [==============================] - 75s 29ms/step - loss: 3.0483 - acc: 0.2438 - val_loss: 2.9807 - val_acc: 0.2231 Epoch 11/100 2600/2600 [==============================] - 76s 29ms/step - loss: 2.9574 - acc: 0.2273 - val_loss: 2.9017 - val_acc: 0.2323 Epoch 12/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.8532 - acc: 0.2419 - val_loss: 2.8193 - val_acc: 0.2385 Epoch 13/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.7798 - acc: 0.2462 - val_loss: 2.7493 - val_acc: 0.2523 Epoch 14/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.7245 - acc: 0.2496 - val_loss: 2.6838 - val_acc: 0.2538 Epoch 15/100 2600/2600 [==============================] - 76s 29ms/step - loss: 2.6775 - acc: 0.2573 - val_loss: 2.6491 - val_acc: 0.2708 Epoch 16/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.6519 - acc: 0.2519 - val_loss: 2.6379 - val_acc: 0.2585 Epoch 17/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.6060 - acc: 0.2585 - val_loss: 2.5812 - val_acc: 0.2585 Epoch 18/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.5590 - acc: 0.2750 - val_loss: 2.5431 - val_acc: 0.2723 Epoch 19/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.5217 - acc: 0.2888 - val_loss: 2.5352 - val_acc: 0.2723 Epoch 20/100 2600/2600 [==============================] - 76s 29ms/step - loss: 2.5025 - acc: 0.2800 - val_loss: 2.5107 - val_acc: 0.2800 Epoch 21/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4854 - acc: 0.2823 - val_loss: 2.5379 - val_acc: 0.2785 Epoch 22/100 2600/2600 [==============================] - 76s 29ms/step - loss: 2.5018 - acc: 0.2919 - val_loss: 2.4890 - val_acc: 0.2677 Epoch 23/100 2600/2600 [==============================] - 76s 29ms/step - loss: 2.4499 - acc: 0.2862 - val_loss: 2.4957 - val_acc: 0.2754 Epoch 24/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4646 - acc: 0.2900 - val_loss: 2.4941 - val_acc: 0.2815 Epoch 25/100 2600/2600 [==============================] - 76s 29ms/step - loss: 2.4458 - acc: 0.2965 - val_loss: 2.5033 - val_acc: 0.2846 Epoch 26/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4284 - acc: 0.3000 - val_loss: 2.5031 - val_acc: 0.2708 Epoch 27/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4354 - acc: 0.3081 - val_loss: 2.4921 - val_acc: 0.2846 Epoch 28/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4178 - acc: 0.3058 - val_loss: 2.5122 - val_acc: 0.2892 Epoch 29/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4286 - acc: 0.3054 - val_loss: 2.5331 - val_acc: 0.3000 Epoch 30/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4177 - acc: 0.3123 - val_loss: 2.4652 - val_acc: 0.2754 Epoch 31/100 2600/2600 [==============================] - 76s 29ms/step - loss: 2.4177 - acc: 0.3088 - val_loss: 2.5128 - val_acc: 0.2846 Epoch 32/100 2600/2600 [==============================] - 75s 29ms/step - loss: 2.4084 - acc: 0.3246 - val_loss: 2.4965 - val_acc: 0.2923 Epoch 33/100 480/2600 [====>.........................] - ETA: 59s - loss: 2.3937 - acc: 0.3000
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-118-17570caf3137> in <module>() 39 second_iteration.compile(opt, scaled_loss, ['accuracy']) 40 ---> 41 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=100, validation_split=0.2, shuffle=False) 42 43 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1*0.1}% (train) vs. {logs["val_acc"]*100//0.1*0.1}% (validation)')
for val in [0.0001, 0.0003, 0.0005, 0.0007, 0.0009]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=40, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=40, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=40, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=40, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=40, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=40, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=150, input_dim=640, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=80, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val * 1.5)))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=13, input_dim=80, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val * 1.5)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=30, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0001 regularization Epoch 1: 8.8% (train) vs. 11.5% (validation) Epoch 2: 12.5% (train) vs. 16.1% (validation) Epoch 3: 16.7% (train) vs. 19.0% (validation) Epoch 4: 19.400000000000002% (train) vs. 21.8% (validation) Epoch 5: 20.700000000000003% (train) vs. 22.900000000000002% (validation) Epoch 6: 20.700000000000003% (train) vs. 23.200000000000003% (validation) Epoch 7: 23.1% (train) vs. 23.5% (validation) Epoch 8: 24.900000000000002% (train) vs. 25.3% (validation) Epoch 9: 24.5% (train) vs. 23.6% (validation) Epoch 10: 28.0% (train) vs. 26.6% (validation) Epoch 11: 28.900000000000002% (train) vs. 26.700000000000003% (validation) Epoch 12: 29.400000000000002% (train) vs. 25.3% (validation) Epoch 13: 30.6% (train) vs. 26.6% (validation) Epoch 14: 30.8% (train) vs. 27.5% (validation) Epoch 15: 34.2% (train) vs. 29.3% (validation) Epoch 16: 34.9% (train) vs. 30.6% (validation) Epoch 17: 36.6% (train) vs. 28.900000000000002% (validation) Epoch 18: 36.0% (train) vs. 28.700000000000003% (validation) Epoch 19: 38.300000000000004% (train) vs. 30.6% (validation) Epoch 20: 37.9% (train) vs. 31.3% (validation) Epoch 21: 40.0% (train) vs. 31.3% (validation) Epoch 22: 42.7% (train) vs. 30.3% (validation) Epoch 23: 44.5% (train) vs. 29.5% (validation) Epoch 24: 47.400000000000006% (train) vs. 29.3% (validation) Epoch 25: 49.300000000000004% (train) vs. 31.0% (validation) Epoch 26: 48.800000000000004% (train) vs. 29.6% (validation) Epoch 27: 50.300000000000004% (train) vs. 31.5% (validation) Epoch 28: 51.6% (train) vs. 31.0% (validation) Epoch 29: 53.800000000000004% (train) vs. 30.1% (validation) Epoch 30: 52.800000000000004% (train) vs. 31.200000000000003% (validation) End: 64.9% (train) vs. 31.200000000000003% (validation) Going 0.0003 regularization Epoch 1: 8.1% (train) vs. 11.0% (validation) Epoch 2: 12.8% (train) vs. 13.0% (validation) Epoch 3: 13.3% (train) vs. 13.600000000000001% (validation) Epoch 4: 13.4% (train) vs. 14.9% (validation) Epoch 5: 13.8% (train) vs. 16.0% (validation) Epoch 6: 15.8% (train) vs. 15.200000000000001% (validation) Epoch 7: 14.8% (train) vs. 16.400000000000002% (validation) Epoch 8: 16.8% (train) vs. 18.3% (validation) Epoch 9: 17.8% (train) vs. 18.7% (validation) Epoch 10: 19.900000000000002% (train) vs. 20.6% (validation) Epoch 11: 20.1% (train) vs. 20.400000000000002% (validation) Epoch 12: 20.8% (train) vs. 21.3% (validation) Epoch 13: 21.8% (train) vs. 21.200000000000003% (validation) Epoch 14: 21.700000000000003% (train) vs. 22.0% (validation) Epoch 15: 21.400000000000002% (train) vs. 23.200000000000003% (validation) Epoch 16: 22.8% (train) vs. 23.0% (validation) Epoch 17: 21.900000000000002% (train) vs. 22.0% (validation) Epoch 18: 23.6% (train) vs. 24.0% (validation) Epoch 19: 23.8% (train) vs. 22.900000000000002% (validation) Epoch 20: 24.6% (train) vs. 24.900000000000002% (validation) Epoch 21: 25.700000000000003% (train) vs. 25.3% (validation) Epoch 22: 26.200000000000003% (train) vs. 25.8% (validation) Epoch 23: 27.200000000000003% (train) vs. 26.700000000000003% (validation) Epoch 24: 25.700000000000003% (train) vs. 26.6% (validation) Epoch 25: 26.8% (train) vs. 25.5% (validation) Epoch 26: 27.3% (train) vs. 27.3% (validation) Epoch 27: 27.6% (train) vs. 28.3% (validation) Epoch 28: 26.700000000000003% (train) vs. 27.3% (validation) Epoch 29: 27.8% (train) vs. 26.900000000000002% (validation) Epoch 30: 28.3% (train) vs. 28.6% (validation) End: 32.1% (train) vs. 28.6% (validation) Going 0.0005 regularization
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-133-dc7968f1ed71> in <module>() 46 cb = [MyCallback()] 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=30, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2945 raise TypeError('`inputs` should be a list or tuple.') 2946 -> 2947 session = get_session() 2948 feed_arrays = [] 2949 array_vals = [] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in get_session() 467 if not _MANUAL_VAR_INIT: 468 with session.graph.as_default(): --> 469 _initialize_variables(session) 470 return session 471 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in _initialize_variables(session) 729 # marked as initialized. 730 is_initialized = session.run( --> 731 [variables_module.is_variable_initialized(v) for v in candidate_vars]) 732 uninitialized_vars = [] 733 for flag, v in zip(is_initialized, candidate_vars): C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in run(self, fetches, feed_dict, options, run_metadata) 927 try: 928 result = self._run(None, fetches, feed_dict, options_ptr, --> 929 run_metadata_ptr) 930 if run_metadata: 931 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in _run(self, handle, fetches, feed_dict, options, run_metadata) 1150 if final_fetches or final_targets or (handle and feed_dict_tensor): 1151 results = self._do_run(handle, final_targets, final_fetches, -> 1152 feed_dict_tensor, options, run_metadata) 1153 else: 1154 results = [] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in _do_run(self, handle, target_list, fetch_list, feed_dict, options, run_metadata) 1326 if handle is None: 1327 return self._do_call(_run_fn, feeds, fetches, targets, options, -> 1328 run_metadata) 1329 else: 1330 return self._do_call(_prun_fn, handle, feeds, fetches) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in _do_call(self, fn, *args) 1332 def _do_call(self, fn, *args): 1333 try: -> 1334 return fn(*args) 1335 except errors.OpError as e: 1336 message = compat.as_text(e.message) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in _run_fn(feed_dict, fetch_list, target_list, options, run_metadata) 1315 def _run_fn(feed_dict, fetch_list, target_list, options, run_metadata): 1316 # Ensure any changes to the graph are reflected in the runtime. -> 1317 self._extend_graph() 1318 return self._call_tf_sessionrun( 1319 options, feed_dict, fetch_list, target_list, run_metadata) C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in _extend_graph(self) 1350 def _extend_graph(self): 1351 with self._graph._session_run_lock(): # pylint: disable=protected-access -> 1352 tf_session.ExtendSession(self._session) 1353 1354 # The threshold to run garbage collection to delete dead tensors. KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=750, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0003 regularization Epoch 1: 9.6% (train) vs. 12.7% (validation) Epoch 2: 15.6% (train) vs. 16.7% (validation) Epoch 3: 17.3% (train) vs. 20.1% (validation) Epoch 4: 18.3% (train) vs. 19.3% (validation) Epoch 5: 21.8% (train) vs. 24.1% (validation) Epoch 6: 23.0% (train) vs. 22.3% (validation) Epoch 7: 24.9% (train) vs. 26.3% (validation) Epoch 8: 24.8% (train) vs. 22.9% (validation) Epoch 9: 27.1% (train) vs. 25.5% (validation) Epoch 10: 28.7% (train) vs. 26.0% (validation) Epoch 11: 28.9% (train) vs. 25.5% (validation) Epoch 12: 31.0% (train) vs. 27.8% (validation) Epoch 13: 30.8% (train) vs. 30.3% (validation) Epoch 14: 34.1% (train) vs. 29.6% (validation) Epoch 15: 34.1% (train) vs. 29.8% (validation) Epoch 16: 35.0% (train) vs. 33.5% (validation) Epoch 17: 36.5% (train) vs. 32.7% (validation) Epoch 18: 38.3% (train) vs. 29.8% (validation) Epoch 19: 37.4% (train) vs. 31.2% (validation) Epoch 20: 39.0% (train) vs. 33.5% (validation) Epoch 21: 40.2% (train) vs. 30.3% (validation) Epoch 22: 42.9% (train) vs. 33.8% (validation) Epoch 23: 44.1% (train) vs. 33.8% (validation) Epoch 24: 45.6% (train) vs. 32.4% (validation) Epoch 25: 47.7% (train) vs. 32.4% (validation) Epoch 26: 48.6% (train) vs. 34.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-137-6de54bc82f47> in <module>() 46 cb = [MyCallback()] 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=750, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0003 regularization Epoch 1: 11.4% (train) vs. 15.9% (validation) Epoch 2: 16.2% (train) vs. 17.6% (validation) Epoch 3: 18.1% (train) vs. 19.5% (validation) Epoch 4: 19.8% (train) vs. 23.2% (validation) Epoch 5: 21.6% (train) vs. 23.0% (validation) Epoch 6: 21.9% (train) vs. 22.3% (validation) Epoch 7: 24.4% (train) vs. 23.2% (validation) Epoch 8: 24.5% (train) vs. 24.4% (validation) Epoch 9: 25.8% (train) vs. 26.3% (validation) Epoch 10: 27.1% (train) vs. 25.0% (validation) Epoch 11: 27.6% (train) vs. 27.8% (validation) Epoch 12: 29.9% (train) vs. 27.0% (validation) Epoch 13: 29.7% (train) vs. 27.5% (validation) Epoch 14: 31.4% (train) vs. 30.6% (validation) Epoch 15: 33.5% (train) vs. 31.0% (validation) Epoch 16: 34.2% (train) vs. 30.4% (validation) Epoch 17: 36.0% (train) vs. 28.7% (validation) Epoch 18: 35.6% (train) vs. 31.0% (validation) Epoch 19: 39.1% (train) vs. 30.7% (validation) Epoch 20: 38.9% (train) vs. 33.3% (validation) Epoch 21: 41.1% (train) vs. 33.5% (validation) Epoch 22: 42.3% (train) vs. 33.6% (validation) Epoch 23: 44.4% (train) vs. 31.6% (validation) Epoch 24: 45.3% (train) vs. 34.1% (validation) Epoch 25: 47.9% (train) vs. 33.0% (validation) Epoch 26: 46.8% (train) vs. 32.3% (validation) Epoch 27: 46.5% (train) vs. 33.8% (validation) Epoch 28: 48.3% (train) vs. 35.2% (validation) Epoch 29: 50.7% (train) vs. 32.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-138-03f58a54646d> in <module>() 46 cb = [MyCallback()] 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=750, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l1(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0003 regularization Epoch 1: 6.9% (train) vs. 7.6% (validation) Epoch 2: 10.8% (train) vs. 13.6% (validation) Epoch 3: 13.9% (train) vs. 17.6% (validation) Epoch 4: 16.2% (train) vs. 17.2% (validation) Epoch 5: 17.5% (train) vs. 18.6% (validation) Epoch 6: 19.6% (train) vs. 21.3% (validation) Epoch 7: 21.0% (train) vs. 23.0% (validation) Epoch 8: 22.1% (train) vs. 22.3% (validation) Epoch 9: 23.1% (train) vs. 24.1% (validation) Epoch 10: 25.0% (train) vs. 27.6% (validation) Epoch 11: 24.7% (train) vs. 24.3% (validation) Epoch 12: 26.2% (train) vs. 23.3% (validation) Epoch 13: 26.1% (train) vs. 23.5% (validation) Epoch 14: 26.4% (train) vs. 23.6% (validation) Epoch 15: 27.7% (train) vs. 27.3% (validation) Epoch 16: 29.5% (train) vs. 28.9% (validation) Epoch 17: 29.2% (train) vs. 29.2% (validation) Epoch 18: 28.7% (train) vs. 29.3% (validation) Epoch 19: 29.5% (train) vs. 28.4% (validation) Epoch 20: 31.3% (train) vs. 29.2% (validation) Epoch 21: 30.6% (train) vs. 29.3% (validation) Epoch 22: 32.4% (train) vs. 31.6% (validation) Epoch 23: 32.9% (train) vs. 30.9% (validation) Epoch 24: 33.3% (train) vs. 31.5% (validation) Epoch 25: 34.3% (train) vs. 32.1% (validation) Epoch 26: 35.9% (train) vs. 33.6% (validation) Epoch 27: 37.2% (train) vs. 33.0% (validation) Epoch 28: 36.3% (train) vs. 33.5% (validation) Epoch 29: 36.6% (train) vs. 32.6% (validation) Epoch 30: 39.4% (train) vs. 32.3% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-139-a0450ea25d3f> in <module>() 46 cb = [MyCallback()] 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=800, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0003 regularization Epoch 1: 7.8% (train) vs. 11.8% (validation) Epoch 2: 12.7% (train) vs. 14.3% (validation) Epoch 3: 15.5% (train) vs. 21.0% (validation) Epoch 4: 20.0% (train) vs. 22.3% (validation) Epoch 5: 21.3% (train) vs. 24.4% (validation) Epoch 6: 21.8% (train) vs. 26.9% (validation) Epoch 7: 24.6% (train) vs. 29.0% (validation) Epoch 8: 27.3% (train) vs. 29.2% (validation) Epoch 9: 27.3% (train) vs. 26.9% (validation) Epoch 10: 30.2% (train) vs. 28.4% (validation) Epoch 11: 31.2% (train) vs. 30.0% (validation) Epoch 12: 33.4% (train) vs. 27.5% (validation) Epoch 13: 34.6% (train) vs. 31.2% (validation) Epoch 14: 35.7% (train) vs. 32.9% (validation) Epoch 15: 38.1% (train) vs. 32.0% (validation) Epoch 16: 39.7% (train) vs. 32.7% (validation) Epoch 17: 41.1% (train) vs. 34.3% (validation) Epoch 18: 45.1% (train) vs. 34.4% (validation) Epoch 19: 46.5% (train) vs. 34.0% (validation) Epoch 20: 50.2% (train) vs. 34.0% (validation) Epoch 21: 48.9% (train) vs. 37.2% (validation) Epoch 22: 51.5% (train) vs. 34.4% (validation) Epoch 23: 54.1% (train) vs. 35.2% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-141-2820929328c3> in <module>() 46 cb = [MyCallback()] 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.002]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=800, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=40, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.002 regularization Epoch 1: 10.5% (train) vs. 11.5% (validation) Epoch 2: 14.3% (train) vs. 18.4% (validation) Epoch 3: 14.7% (train) vs. 10.9% (validation) Epoch 4: 15.8% (train) vs. 20.9% (validation) Epoch 5: 18.4% (train) vs. 22.4% (validation) Epoch 6: 19.0% (train) vs. 24.4% (validation) Epoch 7: 21.6% (train) vs. 24.7% (validation) Epoch 8: 23.6% (train) vs. 26.0% (validation) Epoch 9: 25.1% (train) vs. 28.9% (validation) Epoch 10: 25.7% (train) vs. 27.3% (validation) Epoch 11: 28.1% (train) vs. 29.8% (validation) Epoch 12: 29.0% (train) vs. 33.6% (validation) Epoch 13: 31.6% (train) vs. 33.3% (validation) Epoch 14: 32.6% (train) vs. 32.4% (validation) Epoch 15: 34.1% (train) vs. 32.6% (validation) Epoch 16: 35.1% (train) vs. 28.6% (validation) Epoch 17: 37.0% (train) vs. 33.0% (validation) Epoch 18: 37.1% (train) vs. 33.5% (validation) Epoch 19: 41.6% (train) vs. 31.5% (validation) Epoch 20: 42.7% (train) vs. 32.6% (validation) Epoch 21: 45.7% (train) vs. 34.9% (validation) Epoch 22: 47.3% (train) vs. 34.9% (validation) Epoch 23: 47.1% (train) vs. 30.7% (validation) Epoch 24: 45.9% (train) vs. 33.6% (validation) Epoch 25: 48.7% (train) vs. 32.1% (validation) Epoch 26: 49.8% (train) vs. 33.2% (validation) Epoch 27: 52.9% (train) vs. 33.5% (validation) Epoch 28: 56.0% (train) vs. 34.6% (validation) Epoch 29: 56.7% (train) vs. 32.9% (validation) Epoch 30: 58.7% (train) vs. 34.7% (validation) Epoch 31: 60.7% (train) vs. 32.9% (validation) Epoch 32: 60.6% (train) vs. 33.8% (validation) Epoch 33: 61.8% (train) vs. 33.3% (validation) Epoch 34: 59.3% (train) vs. 34.9% (validation) Epoch 35: 61.9% (train) vs. 31.8% (validation) Epoch 36: 66.0% (train) vs. 30.1% (validation) Epoch 37: 66.3% (train) vs. 33.2% (validation) Epoch 38: 68.8% (train) vs. 32.7% (validation) Epoch 39: 68.1% (train) vs. 34.6% (validation) Epoch 40: 71.9% (train) vs. 34.4% (validation) End: 77.30000000000001% (train) vs. 34.4% (validation)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=50, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=800, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.003 regularization Epoch 1: 7.8% (train) vs. 13.8% (validation) Epoch 2: 11.6% (train) vs. 14.0% (validation) Epoch 3: 15.8% (train) vs. 20.3% (validation) Epoch 4: 18.2% (train) vs. 19.8% (validation) Epoch 5: 19.3% (train) vs. 20.1% (validation) Epoch 6: 20.0% (train) vs. 20.9% (validation) Epoch 7: 22.0% (train) vs. 25.2% (validation) Epoch 8: 22.6% (train) vs. 24.0% (validation) Epoch 9: 24.4% (train) vs. 26.1% (validation) Epoch 10: 24.8% (train) vs. 28.6% (validation) Epoch 11: 26.6% (train) vs. 30.3% (validation) Epoch 12: 27.6% (train) vs. 31.0% (validation) Epoch 13: 28.7% (train) vs. 28.1% (validation) Epoch 14: 30.7% (train) vs. 31.8% (validation) Epoch 15: 30.9% (train) vs. 29.8% (validation) Epoch 16: 34.0% (train) vs. 31.0% (validation) Epoch 17: 32.9% (train) vs. 30.6% (validation) Epoch 18: 35.3% (train) vs. 32.3% (validation) Epoch 19: 35.7% (train) vs. 32.9% (validation) Epoch 20: 37.6% (train) vs. 29.8% (validation) Epoch 21: 38.8% (train) vs. 32.9% (validation) Epoch 22: 41.3% (train) vs. 32.9% (validation) Epoch 23: 43.2% (train) vs. 33.5% (validation) Epoch 24: 42.7% (train) vs. 33.0% (validation) Epoch 25: 45.4% (train) vs. 33.5% (validation) Epoch 26: 44.7% (train) vs. 32.6% (validation) Epoch 27: 48.6% (train) vs. 32.4% (validation) Epoch 28: 50.1% (train) vs. 31.6% (validation) Epoch 29: 49.6% (train) vs. 32.0% (validation) Epoch 30: 52.6% (train) vs. 31.3% (validation) Epoch 31: 53.9% (train) vs. 34.1% (validation) Epoch 32: 54.5% (train) vs. 32.7% (validation) Epoch 33: 56.3% (train) vs. 28.7% (validation) Epoch 34: 55.5% (train) vs. 30.4% (validation) Epoch 35: 56.3% (train) vs. 32.4% (validation) Epoch 36: 57.8% (train) vs. 32.9% (validation) Epoch 37: 60.7% (train) vs. 34.3% (validation) Epoch 38: 63.2% (train) vs. 32.6% (validation) Epoch 39: 64.7% (train) vs. 32.3% (validation) Epoch 40: 62.3% (train) vs. 32.4% (validation) Epoch 41: 64.3% (train) vs. 31.5% (validation) Epoch 42: 65.2% (train) vs. 32.6% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-146-596b72ff93a0> in <module>() 46 cb = [MyCallback()] 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=60, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.002]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.002 regularization Epoch 1: 7.9% (train) vs. 12.6% (validation) Epoch 2: 12.9% (train) vs. 15.3% (validation) Epoch 3: 15.1% (train) vs. 14.6% (validation) Epoch 4: 16.6% (train) vs. 20.7% (validation) Epoch 5: 18.1% (train) vs. 21.3% (validation) Epoch 6: 17.4% (train) vs. 22.3% (validation) Epoch 7: 19.6% (train) vs. 19.8% (validation) Epoch 8: 21.5% (train) vs. 23.3% (validation) Epoch 9: 22.0% (train) vs. 26.9% (validation) Epoch 10: 23.8% (train) vs. 24.4% (validation) Epoch 11: 25.8% (train) vs. 27.6% (validation) Epoch 12: 26.1% (train) vs. 29.0% (validation) Epoch 13: 26.8% (train) vs. 29.2% (validation) Epoch 14: 29.2% (train) vs. 30.6% (validation) Epoch 15: 30.8% (train) vs. 33.8% (validation) Epoch 16: 32.6% (train) vs. 30.7% (validation) Epoch 17: 32.3% (train) vs. 33.2% (validation) Epoch 18: 35.3% (train) vs. 32.4% (validation) Epoch 19: 38.1% (train) vs. 32.7% (validation) Epoch 20: 36.2% (train) vs. 33.6% (validation) Epoch 21: 39.0% (train) vs. 34.6% (validation) Epoch 22: 41.6% (train) vs. 35.3% (validation) Epoch 23: 43.8% (train) vs. 34.7% (validation) Epoch 24: 44.7% (train) vs. 34.9% (validation)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.003 regularization Epoch 1: 9.2% (train) vs. 10.9% (validation) Epoch 2: 12.0% (train) vs. 14.0% (validation) Epoch 3: 13.1% (train) vs. 16.3% (validation) Epoch 4: 15.0% (train) vs. 20.1% (validation) Epoch 5: 16.4% (train) vs. 15.0% (validation) Epoch 6: 17.9% (train) vs. 20.9% (validation) Epoch 7: 18.1% (train) vs. 19.6% (validation) Epoch 8: 19.5% (train) vs. 20.0% (validation) Epoch 9: 22.1% (train) vs. 23.8% (validation) Epoch 10: 22.4% (train) vs. 22.9% (validation) Epoch 11: 23.8% (train) vs. 25.3% (validation) Epoch 12: 23.6% (train) vs. 24.9% (validation) Epoch 13: 25.0% (train) vs. 27.0% (validation) Epoch 14: 26.8% (train) vs. 26.1% (validation) Epoch 15: 26.9% (train) vs. 26.7% (validation) Epoch 16: 26.6% (train) vs. 29.5% (validation) Epoch 17: 28.6% (train) vs. 27.6% (validation) Epoch 18: 28.9% (train) vs. 28.3% (validation) Epoch 19: 30.3% (train) vs. 28.4% (validation) Epoch 20: 31.1% (train) vs. 29.6% (validation) Epoch 21: 33.3% (train) vs. 29.2% (validation)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.002]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.002 regularization Epoch 1: 8.6% (train) vs. 12.6% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-158-0c62caa89bf5> in <module>() 46 cb = [MyCallback()] 47 ---> 48 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 49 50 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.002]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.002 regularization Epoch 1: 6.9% (train) vs. 10.4% (validation) Epoch 2: 9.4% (train) vs. 10.3% (validation) Epoch 3: 10.9% (train) vs. 13.6% (validation) Epoch 4: 11.4% (train) vs. 14.0% (validation) Epoch 5: 13.5% (train) vs. 14.0% (validation) Epoch 6: 13.7% (train) vs. 13.6% (validation) Epoch 7: 14.0% (train) vs. 17.8% (validation) Epoch 8: 13.6% (train) vs. 13.6% (validation) Epoch 9: 14.9% (train) vs. 17.3% (validation) Epoch 10: 13.9% (train) vs. 14.4% (validation) Epoch 11: 13.8% (train) vs. 17.2% (validation) Epoch 12: 14.9% (train) vs. 18.3% (validation) Epoch 13: 14.8% (train) vs. 17.2% (validation) Epoch 14: 15.2% (train) vs. 16.4% (validation) Epoch 15: 15.3% (train) vs. 18.6% (validation) Epoch 16: 16.9% (train) vs. 18.7% (validation) Epoch 17: 16.5% (train) vs. 20.0% (validation) Epoch 18: 18.0% (train) vs. 21.0% (validation) Epoch 19: 17.5% (train) vs. 20.0% (validation) Epoch 20: 18.3% (train) vs. 20.4% (validation) Epoch 21: 18.9% (train) vs. 19.6% (validation) Epoch 22: 18.8% (train) vs. 20.4% (validation) Epoch 23: 18.9% (train) vs. 20.4% (validation) Epoch 24: 19.5% (train) vs. 22.1% (validation) Epoch 25: 19.8% (train) vs. 21.5% (validation) Epoch 26: 19.9% (train) vs. 22.6% (validation) Epoch 27: 21.9% (train) vs. 21.2% (validation) Epoch 28: 22.2% (train) vs. 22.1% (validation) Epoch 29: 22.0% (train) vs. 23.5% (validation) Epoch 30: 23.1% (train) vs. 22.0% (validation) Epoch 31: 23.6% (train) vs. 23.0% (validation) Epoch 32: 23.4% (train) vs. 22.6% (validation) Epoch 33: 23.8% (train) vs. 23.6% (validation) Epoch 34: 23.9% (train) vs. 22.3% (validation) Epoch 35: 25.9% (train) vs. 21.5% (validation) Epoch 36: 24.9% (train) vs. 22.0% (validation) Epoch 37: 25.3% (train) vs. 25.2% (validation) Epoch 38: 26.9% (train) vs. 27.2% (validation) Epoch 39: 26.3% (train) vs. 24.1% (validation) Epoch 40: 26.4% (train) vs. 26.3% (validation) Epoch 41: 26.9% (train) vs. 25.5% (validation) Epoch 42: 28.0% (train) vs. 23.3% (validation) Epoch 43: 28.6% (train) vs. 25.0% (validation) Epoch 44: 27.4% (train) vs. 22.3% (validation) Epoch 45: 29.2% (train) vs. 25.3% (validation) Epoch 46: 27.6% (train) vs. 25.0% (validation) Epoch 47: 29.5% (train) vs. 24.9% (validation) Epoch 48: 30.2% (train) vs. 26.6% (validation) Epoch 49: 29.3% (train) vs. 25.6% (validation) Epoch 50: 31.1% (train) vs. 27.2% (validation) Epoch 51: 31.2% (train) vs. 25.5% (validation) Epoch 52: 30.8% (train) vs. 23.8% (validation) Epoch 53: 30.7% (train) vs. 27.0% (validation) Epoch 54: 32.4% (train) vs. 23.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-160-130614fd8fcd> in <module>() 48 cb = [MyCallback()] 49 ---> 50 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 51 52 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.003, 0.004]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.003 regularization Epoch 1: 12.8% (train) vs. 9.5% (validation) Epoch 2: 15.9% (train) vs. 17.8% (validation) Epoch 3: 18.9% (train) vs. 20.3% (validation) Epoch 4: 21.6% (train) vs. 21.0% (validation) Epoch 5: 23.1% (train) vs. 20.4% (validation) Epoch 6: 24.5% (train) vs. 23.3% (validation) Epoch 7: 25.6% (train) vs. 22.9% (validation) Epoch 8: 25.7% (train) vs. 24.7% (validation) Epoch 9: 27.1% (train) vs. 24.3% (validation) Epoch 10: 28.2% (train) vs. 24.6% (validation) Epoch 11: 30.6% (train) vs. 26.0% (validation) Epoch 12: 32.5% (train) vs. 28.4% (validation) Epoch 13: 34.1% (train) vs. 27.5% (validation) Epoch 14: 35.8% (train) vs. 26.3% (validation) Epoch 15: 36.3% (train) vs. 26.6% (validation) Epoch 16: 37.3% (train) vs. 31.0% (validation) Epoch 17: 38.4% (train) vs. 30.4% (validation) Epoch 18: 40.4% (train) vs. 29.6% (validation) Epoch 19: 41.8% (train) vs. 31.6% (validation) Epoch 20: 43.7% (train) vs. 32.3% (validation) Epoch 21: 45.1% (train) vs. 30.3% (validation) Epoch 22: 44.6% (train) vs. 30.3% (validation) Epoch 23: 51.4% (train) vs. 31.6% (validation) Epoch 24: 54.0% (train) vs. 30.1% (validation) Epoch 25: 50.4% (train) vs. 30.9% (validation) Epoch 26: 54.5% (train) vs. 29.6% (validation) Epoch 27: 54.3% (train) vs. 30.4% (validation) Epoch 28: 56.5% (train) vs. 30.3% (validation) Epoch 29: 57.8% (train) vs. 31.8% (validation) Epoch 30: 63.1% (train) vs. 30.4% (validation) Epoch 31: 64.9% (train) vs. 29.3% (validation) Epoch 32: 64.3% (train) vs. 31.2% (validation) Epoch 33: 65.6% (train) vs. 28.9% (validation) Epoch 34: 68.8% (train) vs. 29.3% (validation) Epoch 35: 68.7% (train) vs. 28.7% (validation) Epoch 36: 68.1% (train) vs. 28.0% (validation) Epoch 37: 69.9% (train) vs. 28.6% (validation) Epoch 38: 71.5% (train) vs. 29.8% (validation) Epoch 39: 73.3% (train) vs. 28.1% (validation) Epoch 40: 71.8% (train) vs. 30.7% (validation) Epoch 41: 76.5% (train) vs. 29.6% (validation) Epoch 42: 77.8% (train) vs. 27.0% (validation) Epoch 43: 79.4% (train) vs. 26.1% (validation) Epoch 44: 77.2% (train) vs. 27.3% (validation) Epoch 45: 78.4% (train) vs. 27.8% (validation) Epoch 46: 80.5% (train) vs. 29.0% (validation) Epoch 47: 82.3% (train) vs. 24.6% (validation) Epoch 48: 81.9% (train) vs. 27.3% (validation) Epoch 49: 85.1% (train) vs. 25.2% (validation) Epoch 50: 87.4% (train) vs. 26.4% (validation) Epoch 51: 88.8% (train) vs. 24.1% (validation) Epoch 52: 87.1% (train) vs. 26.7% (validation) Epoch 53: 88.3% (train) vs. 26.9% (validation) Epoch 54: 87.5% (train) vs. 27.6% (validation) Epoch 55: 88.3% (train) vs. 27.2% (validation) Epoch 56: 89.7% (train) vs. 26.6% (validation) Epoch 57: 91.3% (train) vs. 27.8% (validation) Epoch 58: 92.3% (train) vs. 27.8% (validation) Epoch 59: 90.7% (train) vs. 26.7% (validation) Epoch 60: 92.3% (train) vs. 29.5% (validation) Epoch 61: 90.5% (train) vs. 27.2% (validation) Epoch 62: 90.2% (train) vs. 30.6% (validation) Epoch 63: 93.6% (train) vs. 28.3% (validation) Epoch 64: 94.0% (train) vs. 27.2% (validation) Epoch 65: 93.1% (train) vs. 30.6% (validation) Epoch 66: 93.8% (train) vs. 28.1% (validation) Epoch 67: 94.8% (train) vs. 30.0% (validation) Epoch 68: 96.1% (train) vs. 28.0% (validation) Epoch 69: 94.8% (train) vs. 29.5% (validation) Epoch 70: 92.9% (train) vs. 32.0% (validation) Epoch 71: 93.8% (train) vs. 27.3% (validation) Epoch 72: 93.8% (train) vs. 29.6% (validation) Epoch 73: 93.6% (train) vs. 28.6% (validation) Epoch 74: 94.8% (train) vs. 29.0% (validation) Epoch 75: 95.4% (train) vs. 29.8% (validation) End: 93.0% (train) vs. 29.8% (validation) Going 0.004 regularization Epoch 1: 7.4% (train) vs. 8.4% (validation) Epoch 2: 10.8% (train) vs. 14.9% (validation) Epoch 3: 12.7% (train) vs. 9.6% (validation) Epoch 4: 6.9% (train) vs. 7.8% (validation) Epoch 5: 6.5% (train) vs. 7.5% (validation) Epoch 6: 7.4% (train) vs. 7.5% (validation) Epoch 7: 6.8% (train) vs. 7.5% (validation) Epoch 8: 7.9% (train) vs. 7.6% (validation) Epoch 9: 7.5% (train) vs. 10.4% (validation) Epoch 10: 9.0% (train) vs. 8.1% (validation) Epoch 11: 9.9% (train) vs. 9.6% (validation) Epoch 12: 13.0% (train) vs. 16.4% (validation) Epoch 13: 15.6% (train) vs. 18.7% (validation) Epoch 14: 16.9% (train) vs. 18.3% (validation) Epoch 15: 16.8% (train) vs. 18.1% (validation) Epoch 16: 18.1% (train) vs. 17.9% (validation) Epoch 17: 18.5% (train) vs. 19.6% (validation) Epoch 18: 19.1% (train) vs. 19.2% (validation) Epoch 19: 18.9% (train) vs. 19.0% (validation) Epoch 20: 19.0% (train) vs. 19.2% (validation) Epoch 21: 18.9% (train) vs. 22.0% (validation) Epoch 22: 20.2% (train) vs. 23.3% (validation) Epoch 23: 21.1% (train) vs. 23.5% (validation) Epoch 24: 21.4% (train) vs. 24.4% (validation) Epoch 25: 22.6% (train) vs. 23.3% (validation) Epoch 26: 23.3% (train) vs. 24.6% (validation) Epoch 27: 23.5% (train) vs. 24.1% (validation) Epoch 28: 22.8% (train) vs. 25.6% (validation) Epoch 29: 23.6% (train) vs. 24.1% (validation) Epoch 30: 23.9% (train) vs. 25.8% (validation) Epoch 31: 24.6% (train) vs. 25.2% (validation) Epoch 32: 25.1% (train) vs. 25.5% (validation) Epoch 33: 24.8% (train) vs. 24.6% (validation) Epoch 34: 24.7% (train) vs. 25.0% (validation) Epoch 35: 25.1% (train) vs. 27.0% (validation) Epoch 36: 25.8% (train) vs. 26.4% (validation) Epoch 37: 26.4% (train) vs. 26.9% (validation) Epoch 38: 26.6% (train) vs. 26.7% (validation) Epoch 39: 26.6% (train) vs. 27.5% (validation) Epoch 40: 27.7% (train) vs. 27.6% (validation) Epoch 41: 28.4% (train) vs. 28.0% (validation) Epoch 42: 28.9% (train) vs. 29.0% (validation) Epoch 43: 29.2% (train) vs. 28.7% (validation) Epoch 44: 29.9% (train) vs. 28.3% (validation) Epoch 45: 31.3% (train) vs. 26.6% (validation) Epoch 46: 32.0% (train) vs. 26.3% (validation) Epoch 47: 32.4% (train) vs. 25.5% (validation) Epoch 48: 32.6% (train) vs. 28.0% (validation) Epoch 49: 33.7% (train) vs. 26.4% (validation) Epoch 50: 34.4% (train) vs. 26.3% (validation) Epoch 51: 34.6% (train) vs. 27.8% (validation) Epoch 52: 35.6% (train) vs. 25.8% (validation) Epoch 53: 35.4% (train) vs. 25.5% (validation) Epoch 54: 34.4% (train) vs. 24.9% (validation) Epoch 55: 36.1% (train) vs. 28.1% (validation) Epoch 56: 34.8% (train) vs. 29.0% (validation) Epoch 57: 36.7% (train) vs. 29.5% (validation) Epoch 58: 37.8% (train) vs. 30.3% (validation) Epoch 59: 39.3% (train) vs. 29.0% (validation) Epoch 60: 39.4% (train) vs. 28.6% (validation) Epoch 61: 38.8% (train) vs. 29.2% (validation) Epoch 62: 39.6% (train) vs. 28.6% (validation) Epoch 63: 39.2% (train) vs. 27.0% (validation) Epoch 64: 41.1% (train) vs. 26.6% (validation) Epoch 65: 41.4% (train) vs. 27.2% (validation) Epoch 66: 41.3% (train) vs. 27.6% (validation) Epoch 67: 42.1% (train) vs. 28.0% (validation) Epoch 68: 42.4% (train) vs. 26.9% (validation) Epoch 69: 42.1% (train) vs. 28.3% (validation) Epoch 70: 42.9% (train) vs. 28.4% (validation) Epoch 71: 42.2% (train) vs. 28.0% (validation) Epoch 72: 43.4% (train) vs. 27.8% (validation) Epoch 73: 44.7% (train) vs. 27.5% (validation) Epoch 74: 43.8% (train) vs. 27.0% (validation) Epoch 75: 46.6% (train) vs. 28.1% (validation) End: 41.6% (train) vs. 28.1% (validation)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.001]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 2)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 2)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 2)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val * 2)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=100, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=25, input_dim=50, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=13, input_dim=25, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.001 regularization Epoch 1: 7.9% (train) vs. 8.9% (validation) Epoch 2: 9.8% (train) vs. 12.3% (validation) Epoch 3: 13.1% (train) vs. 16.1% (validation) Epoch 4: 15.1% (train) vs. 17.6% (validation) Epoch 5: 15.3% (train) vs. 18.9% (validation) Epoch 6: 16.8% (train) vs. 17.5% (validation) Epoch 7: 17.9% (train) vs. 19.2% (validation) Epoch 8: 18.3% (train) vs. 17.8% (validation) Epoch 9: 19.3% (train) vs. 19.5% (validation) Epoch 10: 18.5% (train) vs. 20.1% (validation) Epoch 11: 20.3% (train) vs. 21.6% (validation) Epoch 12: 20.0% (train) vs. 22.3% (validation) Epoch 13: 20.9% (train) vs. 21.2% (validation) Epoch 14: 21.1% (train) vs. 22.4% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-180-91369af641af> in <module>() 48 cb = [MyCallback()] 49 ---> 50 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 51 52 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
https://cs231n.github.io/neural-networks-2/#init
Ever initialize the biases to 0.01?
http://www.cs.toronto.edu/~rsalakhu/papers/srivastava14a.pdf
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.001]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=100, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=25, input_dim=50, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=13, input_dim=25, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.001 regularization Epoch 1: 8.1% (train) vs. 7.5% (validation) Epoch 2: 9.3% (train) vs. 11.8% (validation) Epoch 3: 11.2% (train) vs. 14.1% (validation) Epoch 4: 12.2% (train) vs. 14.7% (validation) Epoch 5: 12.4% (train) vs. 15.5% (validation) Epoch 6: 14.1% (train) vs. 14.3% (validation) Epoch 7: 14.3% (train) vs. 16.9% (validation) Epoch 8: 14.5% (train) vs. 16.7% (validation) Epoch 9: 13.6% (train) vs. 15.3% (validation) Epoch 10: 15.6% (train) vs. 15.3% (validation) Epoch 11: 15.3% (train) vs. 10.3% (validation) Epoch 12: 16.2% (train) vs. 17.6% (validation) Epoch 13: 16.5% (train) vs. 16.7% (validation) Epoch 14: 16.6% (train) vs. 20.3% (validation) Epoch 15: 17.7% (train) vs. 19.0% (validation) Epoch 16: 19.1% (train) vs. 18.7% (validation) Epoch 17: 18.6% (train) vs. 19.3% (validation) Epoch 18: 19.3% (train) vs. 21.5% (validation) Epoch 19: 19.3% (train) vs. 21.2% (validation) Epoch 20: 20.5% (train) vs. 23.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-185-ee196e620e6f> in <module>() 48 cb = [MyCallback()] 49 ---> 50 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 51 52 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0001]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.25))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0001 regularization Epoch 1: 9.9% (train) vs. 15.6% (validation) Epoch 2: 11.7% (train) vs. 12.4% (validation) Epoch 3: 14.3% (train) vs. 15.8% (validation) Epoch 4: 15.3% (train) vs. 17.8% (validation) Epoch 5: 17.6% (train) vs. 21.8% (validation) Epoch 6: 20.5% (train) vs. 20.9% (validation) Epoch 7: 21.0% (train) vs. 24.3% (validation) Epoch 8: 22.6% (train) vs. 25.2% (validation) Epoch 9: 23.8% (train) vs. 21.2% (validation) Epoch 10: 23.0% (train) vs. 22.9% (validation) Epoch 11: 26.2% (train) vs. 25.5% (validation) Epoch 12: 26.6% (train) vs. 27.8% (validation) Epoch 13: 27.9% (train) vs. 27.0% (validation) Epoch 14: 29.8% (train) vs. 30.0% (validation) Epoch 15: 31.0% (train) vs. 29.8% (validation) Epoch 16: 32.8% (train) vs. 29.8% (validation) Epoch 17: 35.0% (train) vs. 31.3% (validation) Epoch 18: 36.9% (train) vs. 34.0% (validation) Epoch 19: 38.3% (train) vs. 34.0% (validation) Epoch 20: 40.0% (train) vs. 31.6% (validation) Epoch 21: 42.1% (train) vs. 33.5% (validation) Epoch 22: 43.1% (train) vs. 34.6% (validation) Epoch 23: 43.9% (train) vs. 34.6% (validation) Epoch 24: 45.6% (train) vs. 36.9% (validation) Epoch 25: 48.1% (train) vs. 35.0% (validation) Epoch 26: 51.3% (train) vs. 34.1% (validation) Epoch 27: 50.8% (train) vs. 34.3% (validation) Epoch 28: 53.0% (train) vs. 34.9% (validation) Epoch 29: 53.4% (train) vs. 33.0% (validation) Epoch 30: 54.9% (train) vs. 34.9% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-222-9799a0426d4e> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0003]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0003 regularization Epoch 1: 7.4% (train) vs. 8.0% (validation) Epoch 2: 11.1% (train) vs. 14.9% (validation) Epoch 3: 14.5% (train) vs. 13.6% (validation) Epoch 4: 14.5% (train) vs. 19.2% (validation) Epoch 5: 17.3% (train) vs. 17.2% (validation) Epoch 6: 17.8% (train) vs. 21.6% (validation) Epoch 7: 19.0% (train) vs. 21.6% (validation) Epoch 8: 20.3% (train) vs. 21.2% (validation) Epoch 9: 22.4% (train) vs. 23.8% (validation) Epoch 10: 21.8% (train) vs. 25.8% (validation) Epoch 11: 23.4% (train) vs. 27.0% (validation) Epoch 12: 22.4% (train) vs. 26.3% (validation) Epoch 13: 21.6% (train) vs. 24.0% (validation) Epoch 14: 24.2% (train) vs. 28.1% (validation) Epoch 15: 25.8% (train) vs. 23.8% (validation) Epoch 16: 25.4% (train) vs. 27.0% (validation) Epoch 17: 26.2% (train) vs. 31.8% (validation) Epoch 18: 28.5% (train) vs. 28.9% (validation) Epoch 19: 29.1% (train) vs. 28.6% (validation) Epoch 20: 30.6% (train) vs. 30.7% (validation) Epoch 21: 31.7% (train) vs. 26.4% (validation) Epoch 22: 33.3% (train) vs. 30.6% (validation) Epoch 23: 35.2% (train) vs. 29.6% (validation) Epoch 24: 35.5% (train) vs. 32.1% (validation) Epoch 25: 37.0% (train) vs. 31.2% (validation) Epoch 26: 38.5% (train) vs. 30.9% (validation) Epoch 27: 38.9% (train) vs. 32.3% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-223-e9419dfca3b3> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0002]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/4)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/4)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/4)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/4)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0002 regularization Epoch 1: 7.7% (train) vs. 7.8% (validation) Epoch 2: 10.4% (train) vs. 13.2% (validation) Epoch 3: 12.8% (train) vs. 14.0% (validation) Epoch 4: 14.4% (train) vs. 16.4% (validation) Epoch 5: 15.5% (train) vs. 17.8% (validation) Epoch 6: 17.5% (train) vs. 19.5% (validation) Epoch 7: 19.2% (train) vs. 23.2% (validation) Epoch 8: 18.2% (train) vs. 24.0% (validation) Epoch 9: 21.1% (train) vs. 23.8% (validation) Epoch 10: 23.1% (train) vs. 23.2% (validation) Epoch 11: 24.9% (train) vs. 28.6% (validation) Epoch 12: 24.9% (train) vs. 27.3% (validation) Epoch 13: 26.1% (train) vs. 25.6% (validation) Epoch 14: 26.2% (train) vs. 26.6% (validation) Epoch 15: 26.6% (train) vs. 28.4% (validation) Epoch 16: 28.6% (train) vs. 30.4% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-225-9f68656ca194> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
second_iteration.save('models/model_1.keras')
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=50, input_dim=100, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=25, input_dim=50, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=13, input_dim=25, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0005 regularization Epoch 1: 7.8% (train) vs. 11.0% (validation) Epoch 2: 7.8% (train) vs. 6.0% (validation) Epoch 3: 8.0% (train) vs. 7.8% (validation) Epoch 4: 7.0% (train) vs. 10.7% (validation) Epoch 5: 10.4% (train) vs. 13.0% (validation) Epoch 6: 11.6% (train) vs. 12.7% (validation) Epoch 7: 12.4% (train) vs. 14.9% (validation) Epoch 8: 13.1% (train) vs. 13.5% (validation) Epoch 9: 12.9% (train) vs. 14.6% (validation) Epoch 10: 12.8% (train) vs. 16.1% (validation) Epoch 11: 13.1% (train) vs. 15.0% (validation) Epoch 12: 14.0% (train) vs. 16.1% (validation) Epoch 13: 14.3% (train) vs. 16.9% (validation) Epoch 14: 15.1% (train) vs. 18.0% (validation) Epoch 15: 16.3% (train) vs. 17.6% (validation) Epoch 16: 15.4% (train) vs. 15.0% (validation) Epoch 17: 14.7% (train) vs. 19.5% (validation) Epoch 18: 15.0% (train) vs. 19.3% (validation) Epoch 19: 16.4% (train) vs. 16.9% (validation) Epoch 20: 15.7% (train) vs. 19.6% (validation) Epoch 21: 17.1% (train) vs. 19.0% (validation) Epoch 22: 16.8% (train) vs. 19.0% (validation) Epoch 23: 16.3% (train) vs. 21.0% (validation) Epoch 24: 18.1% (train) vs. 16.3% (validation) Epoch 25: 17.8% (train) vs. 18.4% (validation) Epoch 26: 17.9% (train) vs. 19.2% (validation) Epoch 27: 18.3% (train) vs. 18.3% (validation) Epoch 28: 19.2% (train) vs. 18.3% (validation) Epoch 29: 18.7% (train) vs. 19.5% (validation) Epoch 30: 20.8% (train) vs. 21.2% (validation) Epoch 31: 19.5% (train) vs. 16.9% (validation) Epoch 32: 19.6% (train) vs. 21.5% (validation) Epoch 33: 20.3% (train) vs. 20.4% (validation) Epoch 34: 21.8% (train) vs. 19.5% (validation) Epoch 35: 20.7% (train) vs. 20.4% (validation) Epoch 36: 20.9% (train) vs. 16.3% (validation) Epoch 37: 21.3% (train) vs. 19.5% (validation) Epoch 38: 21.5% (train) vs. 20.3% (validation) Epoch 39: 22.9% (train) vs. 19.9% (validation) Epoch 40: 20.3% (train) vs. 15.2% (validation) Epoch 41: 19.8% (train) vs. 17.0% (validation) Epoch 42: 22.0% (train) vs. 19.2% (validation) Epoch 43: 23.0% (train) vs. 22.4% (validation) Epoch 44: 22.8% (train) vs. 20.3% (validation) Epoch 45: 24.3% (train) vs. 22.7% (validation) Epoch 46: 23.3% (train) vs. 21.6% (validation) Epoch 47: 23.0% (train) vs. 21.8% (validation) Epoch 48: 24.9% (train) vs. 20.6% (validation) Epoch 49: 25.8% (train) vs. 21.3% (validation) Epoch 50: 26.6% (train) vs. 22.4% (validation) Epoch 51: 26.3% (train) vs. 20.3% (validation) Epoch 52: 26.4% (train) vs. 23.0% (validation) Epoch 53: 27.5% (train) vs. 20.7% (validation) Epoch 54: 26.9% (train) vs. 19.2% (validation) Epoch 55: 26.9% (train) vs. 19.8% (validation) Epoch 56: 28.7% (train) vs. 21.2% (validation) Epoch 57: 28.6% (train) vs. 21.5% (validation) Epoch 58: 30.0% (train) vs. 21.6% (validation) Epoch 59: 30.0% (train) vs. 16.7% (validation) Epoch 60: 28.5% (train) vs. 20.9% (validation) Epoch 61: 31.4% (train) vs. 21.3% (validation) Epoch 62: 30.5% (train) vs. 21.8% (validation) Epoch 63: 31.4% (train) vs. 19.9% (validation) Epoch 64: 32.3% (train) vs. 21.8% (validation) Epoch 65: 31.6% (train) vs. 22.4% (validation) Epoch 66: 32.6% (train) vs. 21.0% (validation) Epoch 67: 33.6% (train) vs. 20.6% (validation) Epoch 68: 31.6% (train) vs. 18.9% (validation) Epoch 69: 32.4% (train) vs. 22.9% (validation) Epoch 70: 33.4% (train) vs. 16.6% (validation) Epoch 71: 31.8% (train) vs. 21.8% (validation) Epoch 72: 35.1% (train) vs. 22.0% (validation) Epoch 73: 31.4% (train) vs. 20.6% (validation) Epoch 74: 34.9% (train) vs. 21.6% (validation) Epoch 75: 34.2% (train) vs. 23.2% (validation) End: 42.2% (train) vs. 23.200000000000003% (validation)
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=300, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=150, input_dim=300, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=50, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
second_iteration.add(layers.Dropout(0.5))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros'))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0005 regularization Epoch 1: 8.1% (train) vs. 9.3% (validation) Epoch 2: 7.4% (train) vs. 11.5% (validation) Epoch 3: 10.1% (train) vs. 14.0% (validation) Epoch 4: 11.0% (train) vs. 11.8% (validation) Epoch 5: 11.6% (train) vs. 14.1% (validation) Epoch 6: 13.1% (train) vs. 13.3% (validation) Epoch 7: 12.4% (train) vs. 14.1% (validation) Epoch 8: 13.1% (train) vs. 11.6% (validation) Epoch 9: 13.8% (train) vs. 15.2% (validation) Epoch 10: 14.0% (train) vs. 15.6% (validation) Epoch 11: 14.9% (train) vs. 17.6% (validation) Epoch 12: 15.1% (train) vs. 17.5% (validation) Epoch 13: 17.3% (train) vs. 18.9% (validation) Epoch 14: 18.7% (train) vs. 23.5% (validation) Epoch 15: 18.3% (train) vs. 21.6% (validation) Epoch 16: 18.6% (train) vs. 23.0% (validation) Epoch 17: 20.8% (train) vs. 25.0% (validation) Epoch 18: 22.2% (train) vs. 25.0% (validation) Epoch 19: 22.1% (train) vs. 25.2% (validation) Epoch 20: 23.3% (train) vs. 26.9% (validation) Epoch 21: 23.9% (train) vs. 27.0% (validation) Epoch 22: 24.0% (train) vs. 26.0% (validation) Epoch 23: 23.6% (train) vs. 24.9% (validation) Epoch 24: 25.4% (train) vs. 28.0% (validation) Epoch 25: 22.3% (train) vs. 27.8% (validation) Epoch 26: 24.4% (train) vs. 28.6% (validation) Epoch 27: 26.1% (train) vs. 30.4% (validation) Epoch 28: 27.5% (train) vs. 28.1% (validation) Epoch 29: 29.1% (train) vs. 28.4% (validation) Epoch 30: 27.3% (train) vs. 23.5% (validation) Epoch 31: 27.3% (train) vs. 28.1% (validation) Epoch 32: 29.3% (train) vs. 28.4% (validation) Epoch 33: 31.6% (train) vs. 28.3% (validation) Epoch 34: 31.1% (train) vs. 26.9% (validation) Epoch 35: 32.2% (train) vs. 29.2% (validation) Epoch 36: 32.8% (train) vs. 30.6% (validation) Epoch 37: 33.3% (train) vs. 30.7% (validation) Epoch 38: 34.1% (train) vs. 30.0% (validation) Epoch 39: 34.8% (train) vs. 28.1% (validation) Epoch 40: 35.9% (train) vs. 31.2% (validation) Epoch 41: 37.1% (train) vs. 30.9% (validation) Epoch 42: 38.6% (train) vs. 30.3% (validation) Epoch 43: 38.4% (train) vs. 29.0% (validation) Epoch 44: 39.1% (train) vs. 29.0% (validation) Epoch 45: 38.3% (train) vs. 29.8% (validation) Epoch 46: 41.6% (train) vs. 30.0% (validation) Epoch 47: 40.8% (train) vs. 29.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-244-87dd05d24da4> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
#second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=250, input_dim=1024, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
#second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=125, input_dim=250, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
#second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=50, input_dim=125, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
#second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=50, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 11.0% (train) vs. 15.5% (validation) Epoch 2: 16.5% (train) vs. 20.0% (validation) Epoch 3: 19.8% (train) vs. 22.0% (validation) Epoch 4: 22.2% (train) vs. 23.5% (validation) Epoch 5: 22.7% (train) vs. 24.1% (validation) Epoch 6: 27.5% (train) vs. 27.6% (validation) Epoch 7: 28.4% (train) vs. 26.4% (validation) Epoch 8: 34.1% (train) vs. 31.0% (validation) Epoch 9: 35.6% (train) vs. 30.9% (validation) Epoch 10: 37.0% (train) vs. 29.8% (validation) Epoch 11: 40.9% (train) vs. 32.6% (validation) Epoch 12: 44.6% (train) vs. 34.6% (validation) Epoch 13: 46.7% (train) vs. 32.3% (validation) Epoch 14: 47.3% (train) vs. 32.0% (validation) Epoch 15: 53.2% (train) vs. 32.0% (validation) Epoch 16: 56.0% (train) vs. 32.9% (validation) Epoch 17: 57.1% (train) vs. 33.3% (validation) Epoch 18: 59.4% (train) vs. 33.3% (validation) Epoch 19: 64.3% (train) vs. 31.5% (validation) Epoch 20: 66.0% (train) vs. 27.2% (validation) Epoch 21: 67.0% (train) vs. 31.2% (validation) Epoch 22: 72.7% (train) vs. 32.0% (validation) Epoch 23: 73.7% (train) vs. 29.6% (validation) Epoch 24: 76.6% (train) vs. 29.6% (validation) Epoch 25: 80.2% (train) vs. 31.3% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-248-907b7eadf1d6> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
#second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=150, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
#second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
#second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=35, input_dim=75, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
#second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=35, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 14.1% (train) vs. 16.4% (validation) Epoch 2: 19.2% (train) vs. 18.6% (validation) Epoch 3: 22.3% (train) vs. 22.1% (validation) Epoch 4: 25.0% (train) vs. 23.2% (validation) Epoch 5: 26.7% (train) vs. 21.0% (validation) Epoch 6: 28.1% (train) vs. 25.6% (validation) Epoch 7: 31.0% (train) vs. 28.7% (validation) Epoch 8: 32.6% (train) vs. 28.7% (validation) Epoch 9: 34.9% (train) vs. 29.2% (validation) Epoch 10: 36.5% (train) vs. 29.5% (validation) Epoch 11: 40.5% (train) vs. 32.4% (validation) Epoch 12: 44.5% (train) vs. 29.8% (validation) Epoch 13: 47.4% (train) vs. 30.9% (validation) Epoch 14: 50.2% (train) vs. 30.9% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-249-c07b8ff8d498> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=150, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=35, input_dim=75, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=35, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 7.2% (train) vs. 8.3% (validation) Epoch 2: 7.5% (train) vs. 8.4% (validation) Epoch 3: 8.6% (train) vs. 8.4% (validation) Epoch 4: 11.1% (train) vs. 12.7% (validation) Epoch 5: 13.3% (train) vs. 15.3% (validation) Epoch 6: 14.7% (train) vs. 17.3% (validation) Epoch 7: 15.7% (train) vs. 19.2% (validation) Epoch 8: 17.3% (train) vs. 17.0% (validation) Epoch 9: 17.7% (train) vs. 19.2% (validation) Epoch 10: 18.3% (train) vs. 20.7% (validation) Epoch 11: 18.4% (train) vs. 20.3% (validation) Epoch 12: 20.9% (train) vs. 22.0% (validation) Epoch 13: 20.9% (train) vs. 19.6% (validation) Epoch 14: 21.7% (train) vs. 23.0% (validation) Epoch 15: 23.1% (train) vs. 23.3% (validation) Epoch 16: 22.3% (train) vs. 20.6% (validation) Epoch 17: 24.2% (train) vs. 25.6% (validation) Epoch 18: 24.5% (train) vs. 26.7% (validation) Epoch 19: 26.7% (train) vs. 22.9% (validation) Epoch 20: 26.7% (train) vs. 26.3% (validation) Epoch 21: 28.2% (train) vs. 27.2% (validation) Epoch 22: 29.7% (train) vs. 25.6% (validation) Epoch 23: 29.9% (train) vs. 26.4% (validation) Epoch 24: 29.1% (train) vs. 24.9% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-250-bd7c0e9582a8> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=150, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=35, input_dim=75, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=35, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0005 regularization Epoch 1: 7.3% (train) vs. 9.6% (validation) Epoch 2: 9.8% (train) vs. 11.8% (validation) Epoch 3: 13.3% (train) vs. 11.8% (validation) Epoch 4: 15.3% (train) vs. 18.7% (validation) Epoch 5: 16.6% (train) vs. 20.7% (validation) Epoch 6: 18.4% (train) vs. 17.0% (validation) Epoch 7: 18.4% (train) vs. 20.7% (validation) Epoch 8: 20.1% (train) vs. 19.5% (validation) Epoch 9: 19.4% (train) vs. 21.3% (validation) Epoch 10: 20.2% (train) vs. 20.4% (validation) Epoch 11: 20.9% (train) vs. 23.6% (validation) Epoch 12: 22.9% (train) vs. 23.0% (validation) Epoch 13: 23.6% (train) vs. 24.1% (validation) Epoch 14: 23.3% (train) vs. 23.5% (validation) Epoch 15: 24.6% (train) vs. 25.2% (validation) Epoch 16: 24.1% (train) vs. 24.7% (validation) Epoch 17: 25.8% (train) vs. 24.9% (validation) Epoch 18: 25.3% (train) vs. 25.6% (validation) Epoch 19: 26.4% (train) vs. 25.8% (validation) Epoch 20: 26.6% (train) vs. 25.0% (validation) Epoch 21: 27.2% (train) vs. 27.8% (validation) Epoch 22: 27.8% (train) vs. 27.6% (validation) Epoch 23: 28.3% (train) vs. 27.5% (validation) Epoch 24: 30.4% (train) vs. 29.2% (validation) Epoch 25: 30.9% (train) vs. 28.9% (validation) Epoch 26: 30.2% (train) vs. 29.0% (validation) Epoch 27: 33.6% (train) vs. 28.1% (validation) Epoch 28: 32.9% (train) vs. 28.9% (validation) Epoch 29: 33.4% (train) vs. 27.5% (validation) Epoch 30: 34.3% (train) vs. 28.4% (validation) Epoch 31: 36.0% (train) vs. 28.0% (validation) Epoch 32: 36.9% (train) vs. 30.0% (validation) Epoch 33: 38.4% (train) vs. 29.3% (validation) Epoch 34: 37.5% (train) vs. 27.5% (validation) Epoch 35: 40.6% (train) vs. 28.1% (validation) Epoch 36: 39.3% (train) vs. 26.9% (validation) Epoch 37: 40.4% (train) vs. 28.7% (validation) Epoch 38: 42.4% (train) vs. 26.4% (validation) Epoch 39: 37.8% (train) vs. 29.2% (validation) Epoch 40: 42.0% (train) vs. 29.2% (validation) Epoch 41: 42.9% (train) vs. 26.0% (validation) Epoch 42: 43.9% (train) vs. 27.6% (validation) Epoch 43: 42.5% (train) vs. 25.3% (validation) Epoch 44: 44.5% (train) vs. 29.5% (validation) Epoch 45: 45.1% (train) vs. 26.9% (validation) Epoch 46: 44.3% (train) vs. 28.0% (validation) Epoch 47: 48.5% (train) vs. 29.8% (validation) Epoch 48: 46.6% (train) vs. 30.9% (validation) Epoch 49: 49.9% (train) vs. 28.6% (validation) Epoch 50: 52.4% (train) vs. 27.6% (validation) Epoch 51: 50.8% (train) vs. 28.9% (validation) Epoch 52: 49.7% (train) vs. 27.2% (validation) Epoch 53: 51.1% (train) vs. 25.9% (validation) Epoch 54: 49.4% (train) vs. 27.6% (validation) Epoch 55: 51.1% (train) vs. 29.0% (validation) Epoch 56: 53.1% (train) vs. 29.0% (validation) Epoch 57: 53.3% (train) vs. 30.9% (validation) Epoch 58: 54.2% (train) vs. 29.2% (validation) Epoch 59: 55.9% (train) vs. 28.3% (validation) Epoch 60: 56.1% (train) vs. 28.9% (validation) Epoch 61: 53.3% (train) vs. 26.4% (validation) Epoch 62: 57.1% (train) vs. 26.0% (validation) Epoch 63: 57.4% (train) vs. 29.2% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-252-b6a92409d89f> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2969 tensor_type = dtypes_module.as_dtype(tensor.dtype) 2970 array_vals.append(np.asarray(value, -> 2971 dtype=tensor_type.as_numpy_dtype)) 2972 2973 if self.feed_dict: C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\numpy\core\numeric.py in asarray(a, dtype, order) 499 500 """ --> 501 return array(a, dtype, copy=False, order=order) 502 503 KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10)): cce = tf.multiply(cce, 1.1)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.001]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=150, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/2)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/2)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=35, input_dim=75, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/2)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=35, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/2)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.001 regularization Epoch 1: 6.5% (train) vs. 8.1% (validation) Epoch 2: 7.6% (train) vs. 12.4% (validation) Epoch 3: 10.8% (train) vs. 13.5% (validation) Epoch 4: 12.7% (train) vs. 15.3% (validation) Epoch 5: 13.7% (train) vs. 14.1% (validation) Epoch 6: 15.1% (train) vs. 17.0% (validation) Epoch 7: 15.9% (train) vs. 17.0% (validation) Epoch 8: 16.7% (train) vs. 19.0% (validation) Epoch 9: 19.2% (train) vs. 19.8% (validation) Epoch 10: 21.4% (train) vs. 21.8% (validation) Epoch 11: 19.3% (train) vs. 23.0% (validation) Epoch 12: 21.0% (train) vs. 23.0% (validation) Epoch 13: 21.8% (train) vs. 28.0% (validation) Epoch 14: 22.2% (train) vs. 22.7% (validation) Epoch 15: 22.8% (train) vs. 24.0% (validation) Epoch 16: 22.4% (train) vs. 26.7% (validation) Epoch 17: 24.0% (train) vs. 23.5% (validation) Epoch 18: 23.8% (train) vs. 25.6% (validation) Epoch 19: 24.4% (train) vs. 27.0% (validation) Epoch 20: 25.2% (train) vs. 26.3% (validation) Epoch 21: 25.6% (train) vs. 26.3% (validation) Epoch 22: 26.9% (train) vs. 26.7% (validation) Epoch 23: 26.3% (train) vs. 25.3% (validation) Epoch 24: 27.5% (train) vs. 25.2% (validation) Epoch 25: 28.3% (train) vs. 26.7% (validation) Epoch 26: 28.4% (train) vs. 25.0% (validation) Epoch 27: 28.1% (train) vs. 26.1% (validation) Epoch 28: 29.6% (train) vs. 27.2% (validation) Epoch 29: 29.1% (train) vs. 28.9% (validation) Epoch 30: 29.4% (train) vs. 25.5% (validation) Epoch 31: 31.2% (train) vs. 25.5% (validation) Epoch 32: 31.1% (train) vs. 25.0% (validation) Epoch 33: 31.8% (train) vs. 25.3% (validation) Epoch 34: 32.7% (train) vs. 26.7% (validation) Epoch 35: 32.6% (train) vs. 26.4% (validation) Epoch 36: 33.1% (train) vs. 25.8% (validation) Epoch 37: 33.4% (train) vs. 24.9% (validation) Epoch 38: 34.6% (train) vs. 27.8% (validation) Epoch 39: 35.3% (train) vs. 29.0% (validation) Epoch 40: 34.1% (train) vs. 25.6% (validation) Epoch 41: 36.7% (train) vs. 28.7% (validation) Epoch 42: 36.3% (train) vs. 28.0% (validation) Epoch 43: 37.7% (train) vs. 26.7% (validation) Epoch 44: 37.4% (train) vs. 26.0% (validation) Epoch 45: 37.7% (train) vs. 28.9% (validation) Epoch 46: 37.9% (train) vs. 24.3% (validation) Epoch 47: 41.4% (train) vs. 26.0% (validation) Epoch 48: 41.1% (train) vs. 25.3% (validation) Epoch 49: 40.0% (train) vs. 28.6% (validation) Epoch 50: 39.2% (train) vs. 25.0% (validation) Epoch 51: 40.8% (train) vs. 23.8% (validation) Epoch 52: 40.3% (train) vs. 27.6% (validation) Epoch 53: 41.9% (train) vs. 29.0% (validation) Epoch 54: 44.1% (train) vs. 26.7% (validation) Epoch 55: 41.2% (train) vs. 26.7% (validation) Epoch 56: 43.7% (train) vs. 27.8% (validation) Epoch 57: 44.1% (train) vs. 25.6% (validation) Epoch 58: 45.2% (train) vs. 22.7% (validation) Epoch 59: 41.6% (train) vs. 23.6% (validation) Epoch 60: 42.2% (train) vs. 24.3% (validation) Epoch 61: 44.0% (train) vs. 24.7% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-254-32d879f1ce4c> in <module>() 49 cb = [MyCallback()] 50 ---> 51 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 52 53 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)): cce = tf.multiply(cce, 1.2)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.002]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=150, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.0001)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=75, input_dim=150, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.0001)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=35, input_dim=75, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.0001)))
second_iteration.add(layers.Dropout(0.4))
second_iteration.add(layers.Dense(units=13, input_dim=35, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.0001)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.002 regularization Epoch 1: 8.3% (train) vs. 12.1% (validation) Epoch 2: 11.1% (train) vs. 13.3% (validation) Epoch 3: 12.4% (train) vs. 13.2% (validation) Epoch 4: 13.2% (train) vs. 13.6% (validation) Epoch 5: 14.1% (train) vs. 14.4% (validation) Epoch 6: 14.7% (train) vs. 16.6% (validation) Epoch 7: 16.6% (train) vs. 18.0% (validation) Epoch 8: 16.9% (train) vs. 20.1% (validation) Epoch 9: 18.4% (train) vs. 20.6% (validation) Epoch 10: 18.1% (train) vs. 19.3% (validation) Epoch 11: 19.4% (train) vs. 24.3% (validation) Epoch 12: 20.9% (train) vs. 22.7% (validation) Epoch 13: 20.1% (train) vs. 22.7% (validation) Epoch 14: 20.1% (train) vs. 22.0% (validation) Epoch 15: 21.7% (train) vs. 22.6% (validation) Epoch 16: 23.4% (train) vs. 22.6% (validation) Epoch 17: 23.0% (train) vs. 21.8% (validation) Epoch 18: 23.6% (train) vs. 23.8% (validation) Epoch 19: 23.6% (train) vs. 21.5% (validation) Epoch 20: 25.4% (train) vs. 24.3% (validation) Epoch 21: 24.7% (train) vs. 21.3% (validation) Epoch 22: 25.8% (train) vs. 24.4% (validation) Epoch 23: 26.6% (train) vs. 23.9% (validation) Epoch 24: 27.0% (train) vs. 25.2% (validation) Epoch 25: 28.9% (train) vs. 26.3% (validation) Epoch 26: 27.5% (train) vs. 25.5% (validation) Epoch 27: 29.0% (train) vs. 26.6% (validation) Epoch 28: 30.8% (train) vs. 25.2% (validation) Epoch 29: 29.8% (train) vs. 25.5% (validation) Epoch 30: 30.2% (train) vs. 23.3% (validation) Epoch 31: 30.8% (train) vs. 22.3% (validation) Epoch 32: 30.7% (train) vs. 25.3% (validation) Epoch 33: 32.0% (train) vs. 25.8% (validation) Epoch 34: 33.8% (train) vs. 25.3% (validation) Epoch 35: 33.1% (train) vs. 25.8% (validation) Epoch 36: 35.6% (train) vs. 26.3% (validation) Epoch 37: 36.4% (train) vs. 26.7% (validation) Epoch 38: 35.1% (train) vs. 26.6% (validation) Epoch 39: 38.1% (train) vs. 25.0% (validation) Epoch 40: 37.9% (train) vs. 26.0% (validation) Epoch 41: 40.7% (train) vs. 25.8% (validation) Epoch 42: 39.5% (train) vs. 25.5% (validation) Epoch 43: 41.8% (train) vs. 26.4% (validation) Epoch 44: 43.1% (train) vs. 24.6% (validation) Epoch 45: 41.6% (train) vs. 26.4% (validation) Epoch 46: 42.8% (train) vs. 26.6% (validation) Epoch 47: 45.6% (train) vs. 25.5% (validation) Epoch 48: 43.9% (train) vs. 26.3% (validation) Epoch 49: 44.6% (train) vs. 24.3% (validation) Epoch 50: 48.0% (train) vs. 26.9% (validation) Epoch 51: 46.9% (train) vs. 26.9% (validation) Epoch 52: 48.9% (train) vs. 25.6% (validation) Epoch 53: 49.8% (train) vs. 25.2% (validation) Epoch 54: 50.6% (train) vs. 26.0% (validation) Epoch 55: 49.8% (train) vs. 26.4% (validation) Epoch 56: 52.4% (train) vs. 27.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-22-de1402c5294f> in <module>() 52 cb = [MyCallback()] 53 ---> 54 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 55 56 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if (chosen, y_pred) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)): cce = tf.multiply(cce, 1.2)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=200, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.0001)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.0001)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0.0001)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, scaled_loss, ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 8.4% (train) vs. 11.6% (validation) Epoch 2: 10.3% (train) vs. 8.7% (validation) Epoch 3: 13.1% (train) vs. 15.5% (validation) Epoch 4: 16.6% (train) vs. 18.6% (validation) Epoch 5: 17.5% (train) vs. 19.0% (validation) Epoch 6: 20.0% (train) vs. 19.2% (validation) Epoch 7: 21.0% (train) vs. 22.3% (validation) Epoch 8: 23.5% (train) vs. 25.3% (validation) Epoch 9: 25.8% (train) vs. 25.3% (validation) Epoch 10: 27.2% (train) vs. 26.1% (validation) Epoch 11: 27.9% (train) vs. 27.2% (validation) Epoch 12: 28.7% (train) vs. 27.5% (validation) Epoch 13: 31.0% (train) vs. 27.0% (validation) Epoch 14: 30.1% (train) vs. 26.7% (validation) Epoch 15: 32.1% (train) vs. 28.4% (validation) Epoch 16: 34.6% (train) vs. 28.7% (validation) Epoch 17: 38.0% (train) vs. 27.5% (validation) Epoch 18: 36.9% (train) vs. 29.3% (validation) Epoch 19: 39.4% (train) vs. 28.7% (validation) Epoch 20: 40.1% (train) vs. 28.6% (validation) Epoch 21: 40.7% (train) vs. 28.4% (validation) Epoch 22: 43.8% (train) vs. 25.0% (validation) Epoch 23: 47.4% (train) vs. 25.5% (validation) Epoch 24: 48.1% (train) vs. 26.1% (validation) Epoch 25: 47.4% (train) vs. 26.7% (validation) Epoch 26: 51.4% (train) vs. 24.3% (validation) Epoch 27: 52.7% (train) vs. 26.0% (validation) Epoch 28: 54.6% (train) vs. 24.4% (validation) Epoch 29: 55.4% (train) vs. 21.5% (validation) Epoch 30: 56.2% (train) vs. 26.4% (validation) Epoch 31: 56.6% (train) vs. 28.7% (validation) Epoch 32: 58.4% (train) vs. 29.2% (validation) Epoch 33: 59.9% (train) vs. 29.0% (validation) Epoch 34: 60.4% (train) vs. 26.9% (validation) Epoch 35: 63.7% (train) vs. 29.9% (validation) Epoch 36: 63.0% (train) vs. 28.6% (validation) Epoch 37: 56.5% (train) vs. 28.6% (validation) Epoch 38: 62.0% (train) vs. 28.6% (validation) Epoch 39: 65.8% (train) vs. 28.3% (validation) Epoch 40: 69.0% (train) vs. 27.3% (validation) Epoch 41: 69.8% (train) vs. 26.0% (validation) Epoch 42: 69.4% (train) vs. 26.3% (validation) Epoch 43: 71.2% (train) vs. 30.3% (validation) Epoch 44: 75.4% (train) vs. 28.9% (validation) Epoch 45: 72.4% (train) vs. 25.6% (validation) Epoch 46: 76.0% (train) vs. 27.0% (validation) Epoch 47: 76.9% (train) vs. 28.9% (validation) Epoch 48: 77.4% (train) vs. 26.3% (validation) Epoch 49: 76.3% (train) vs. 25.0% (validation) Epoch 50: 78.8% (train) vs. 28.7% (validation) Epoch 51: 78.7% (train) vs. 27.3% (validation) Epoch 52: 80.0% (train) vs. 26.4% (validation) Epoch 53: 79.9% (train) vs. 27.2% (validation) Epoch 54: 82.3% (train) vs. 28.1% (validation) Epoch 55: 82.5% (train) vs. 25.8% (validation) Epoch 56: 80.1% (train) vs. 27.8% (validation) Epoch 57: 83.4% (train) vs. 27.6% (validation) Epoch 58: 83.3% (train) vs. 25.6% (validation) Epoch 59: 85.1% (train) vs. 26.6% (validation) Epoch 60: 84.2% (train) vs. 25.3% (validation) Epoch 61: 84.1% (train) vs. 27.5% (validation) Epoch 62: 85.4% (train) vs. 28.0% (validation) Epoch 63: 85.4% (train) vs. 28.6% (validation) Epoch 64: 83.2% (train) vs. 27.8% (validation) Epoch 65: 80.1% (train) vs. 29.2% (validation) Epoch 66: 86.5% (train) vs. 26.9% (validation) Epoch 67: 84.9% (train) vs. 28.9% (validation) Epoch 68: 84.7% (train) vs. 28.0% (validation) Epoch 69: 86.1% (train) vs. 26.7% (validation) Epoch 70: 85.8% (train) vs. 28.0% (validation) Epoch 71: 86.9% (train) vs. 28.1% (validation) Epoch 72: 86.3% (train) vs. 26.3% (validation) Epoch 73: 85.0% (train) vs. 26.1% (validation) Epoch 74: 84.8% (train) vs. 26.6% (validation) Epoch 75: 88.1% (train) vs. 27.2% (validation) End: 94.80000000000001% (train) vs. 27.200000000000003% (validation)
with tf.Session() as sess:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=200, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.005 regularization Epoch 1: 9.3% (train) vs. 14.4% (validation) Epoch 2: 13.6% (train) vs. 15.6% (validation) Epoch 3: 16.1% (train) vs. 17.8% (validation) Epoch 4: 17.5% (train) vs. 20.3% (validation) Epoch 5: 18.0% (train) vs. 20.7% (validation) Epoch 6: 18.9% (train) vs. 20.6% (validation) Epoch 7: 20.8% (train) vs. 21.3% (validation) Epoch 8: 20.1% (train) vs. 22.0% (validation) Epoch 9: 22.3% (train) vs. 19.8% (validation) Epoch 10: 21.1% (train) vs. 22.1% (validation) Epoch 11: 22.9% (train) vs. 21.2% (validation) Epoch 12: 23.6% (train) vs. 21.2% (validation) Epoch 13: 25.4% (train) vs. 22.7% (validation) Epoch 14: 25.5% (train) vs. 24.9% (validation) Epoch 15: 27.3% (train) vs. 23.8% (validation) Epoch 16: 27.6% (train) vs. 25.8% (validation) Epoch 17: 28.0% (train) vs. 27.2% (validation) Epoch 18: 28.5% (train) vs. 24.9% (validation) Epoch 19: 31.3% (train) vs. 25.8% (validation) Epoch 20: 31.9% (train) vs. 29.3% (validation) Epoch 21: 33.1% (train) vs. 31.6% (validation) Epoch 22: 32.0% (train) vs. 27.8% (validation) Epoch 23: 32.9% (train) vs. 29.6% (validation) Epoch 24: 33.4% (train) vs. 29.2% (validation) Epoch 25: 34.9% (train) vs. 30.3% (validation) Epoch 26: 34.5% (train) vs. 30.6% (validation) Epoch 27: 36.1% (train) vs. 30.4% (validation) Epoch 28: 35.6% (train) vs. 30.6% (validation) Epoch 29: 38.5% (train) vs. 31.3% (validation) Epoch 30: 37.1% (train) vs. 28.7% (validation) Epoch 31: 38.1% (train) vs. 31.2% (validation) Epoch 32: 39.8% (train) vs. 29.2% (validation) Epoch 33: 39.1% (train) vs. 30.3% (validation) Epoch 34: 37.6% (train) vs. 30.4% (validation) Epoch 35: 38.4% (train) vs. 29.3% (validation) Epoch 36: 40.5% (train) vs. 30.7% (validation) Epoch 37: 42.0% (train) vs. 29.8% (validation) Epoch 38: 44.4% (train) vs. 27.5% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-92-c43daf0196e4> in <module>() 55 cb = [MyCallback()] 56 ---> 57 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 58 59 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
with tf.Session() as sess:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=200, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/10)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/10)))
second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val/10)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.005 regularization Epoch 1: 9.3% (train) vs. 12.3% (validation) Epoch 2: 12.0% (train) vs. 13.6% (validation) Epoch 3: 12.3% (train) vs. 13.5% (validation) Epoch 4: 15.4% (train) vs. 16.7% (validation) Epoch 5: 15.9% (train) vs. 17.3% (validation) Epoch 6: 17.1% (train) vs. 19.6% (validation) Epoch 7: 16.9% (train) vs. 21.0% (validation) Epoch 8: 18.7% (train) vs. 20.1% (validation) Epoch 9: 20.5% (train) vs. 22.1% (validation) Epoch 10: 20.6% (train) vs. 22.0% (validation) Epoch 11: 21.2% (train) vs. 22.6% (validation) Epoch 12: 22.0% (train) vs. 23.5% (validation) Epoch 13: 21.9% (train) vs. 26.0% (validation) Epoch 14: 22.9% (train) vs. 25.6% (validation) Epoch 15: 23.6% (train) vs. 24.9% (validation) Epoch 16: 24.9% (train) vs. 25.6% (validation) Epoch 17: 25.9% (train) vs. 23.5% (validation) Epoch 18: 24.5% (train) vs. 23.6% (validation) Epoch 19: 26.7% (train) vs. 23.3% (validation) Epoch 20: 25.8% (train) vs. 23.5% (validation) Epoch 21: 27.5% (train) vs. 21.2% (validation) Epoch 22: 27.4% (train) vs. 22.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-94-b99690663213> in <module>() 55 cb = [MyCallback()] 56 ---> 57 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 58 59 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
https://medium.com/geekculture/how-does-batch-size-impact-your-model-learning-2dd34d9fb1fa
The batch size is another hyperparameter that could be tuned. According to the article above, decreased batch sizes often result in better generalization. And it's not too difficult to see why, since lower batch size means that a network doesn't need to compute the gradient as accurately.
Let's try a network with no dense regularization but a batch size of 16 instead of 32.
with tf.Session() as sess:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
#second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=200, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
#second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
#second_iteration.add(layers.Dropout(0.3))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(0)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 8.8% (train) vs. 11.6% (validation) Epoch 2: 14.0% (train) vs. 12.1% (validation) Epoch 3: 15.3% (train) vs. 11.8% (validation) Epoch 4: 17.6% (train) vs. 15.3% (validation) Epoch 5: 19.8% (train) vs. 20.0% (validation) Epoch 6: 22.3% (train) vs. 15.6% (validation) Epoch 7: 23.0% (train) vs. 24.7% (validation) Epoch 8: 26.3% (train) vs. 23.6% (validation) Epoch 9: 28.6% (train) vs. 25.5% (validation) Epoch 10: 30.8% (train) vs. 25.0% (validation) Epoch 11: 32.0% (train) vs. 26.3% (validation) Epoch 12: 35.3% (train) vs. 24.3% (validation) Epoch 13: 38.5% (train) vs. 26.3% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-96-bd479ca853c9> in <module>() 55 cb = [MyCallback()] 56 ---> 57 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 58 59 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
with tf.Session() as sess:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.001]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=200, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.001 regularization Epoch 1: 9.2% (train) vs. 12.0% (validation) Epoch 2: 11.9% (train) vs. 12.3% (validation) Epoch 3: 12.4% (train) vs. 14.3% (validation) Epoch 4: 13.2% (train) vs. 15.5% (validation) Epoch 5: 14.8% (train) vs. 17.3% (validation) Epoch 6: 16.8% (train) vs. 18.1% (validation) Epoch 7: 17.4% (train) vs. 20.0% (validation) Epoch 8: 19.5% (train) vs. 20.7% (validation) Epoch 9: 20.8% (train) vs. 22.0% (validation) Epoch 10: 22.8% (train) vs. 24.1% (validation) Epoch 11: 23.1% (train) vs. 24.3% (validation) Epoch 12: 24.0% (train) vs. 24.7% (validation) Epoch 13: 24.0% (train) vs. 24.7% (validation) Epoch 14: 24.7% (train) vs. 23.0% (validation) Epoch 15: 25.0% (train) vs. 23.8% (validation) Epoch 16: 25.8% (train) vs. 24.4% (validation) Epoch 17: 26.5% (train) vs. 24.6% (validation) Epoch 18: 26.6% (train) vs. 24.3% (validation) Epoch 19: 27.5% (train) vs. 22.0% (validation) Epoch 20: 28.6% (train) vs. 21.6% (validation) Epoch 21: 27.8% (train) vs. 22.9% (validation) Epoch 22: 29.6% (train) vs. 26.4% (validation) Epoch 23: 30.4% (train) vs. 23.5% (validation) Epoch 24: 31.9% (train) vs. 23.6% (validation) Epoch 25: 32.2% (train) vs. 22.1% (validation) Epoch 26: 34.3% (train) vs. 23.5% (validation) Epoch 27: 32.7% (train) vs. 24.9% (validation) Epoch 28: 34.4% (train) vs. 23.6% (validation) Epoch 29: 35.3% (train) vs. 21.5% (validation) Epoch 30: 37.7% (train) vs. 21.8% (validation) Epoch 31: 37.6% (train) vs. 23.0% (validation) Epoch 32: 40.2% (train) vs. 23.8% (validation) Epoch 33: 40.6% (train) vs. 24.3% (validation) Epoch 34: 43.6% (train) vs. 21.2% (validation) Epoch 35: 43.1% (train) vs. 22.9% (validation) Epoch 36: 45.3% (train) vs. 24.0% (validation) Epoch 37: 44.6% (train) vs. 25.8% (validation) Epoch 38: 45.9% (train) vs. 23.6% (validation) Epoch 39: 50.1% (train) vs. 22.0% (validation) Epoch 40: 50.0% (train) vs. 25.0% (validation) Epoch 41: 52.2% (train) vs. 23.5% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-99-a0cccfd24dac> in <module>() 55 cb = [MyCallback()] 56 ---> 57 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 58 59 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
with tf.Session() as sess:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0, 0.0005]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=80, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=40, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=13, input_dim=40, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 7.3% (train) vs. 8.0% (validation) Epoch 2: 7.7% (train) vs. 8.3% (validation) Epoch 3: 10.1% (train) vs. 10.7% (validation) Epoch 4: 9.8% (train) vs. 11.2% (validation) Epoch 5: 12.9% (train) vs. 17.2% (validation) Epoch 6: 17.6% (train) vs. 19.5% (validation) Epoch 7: 18.5% (train) vs. 18.7% (validation) Epoch 8: 20.8% (train) vs. 20.6% (validation) Epoch 9: 22.7% (train) vs. 19.8% (validation) Epoch 10: 24.7% (train) vs. 22.6% (validation) Epoch 11: 24.3% (train) vs. 23.6% (validation) Epoch 12: 27.0% (train) vs. 24.6% (validation) Epoch 13: 27.1% (train) vs. 25.2% (validation) Epoch 14: 30.0% (train) vs. 25.8% (validation) Epoch 15: 31.1% (train) vs. 28.4% (validation) Epoch 16: 32.7% (train) vs. 27.2% (validation) Epoch 17: 36.0% (train) vs. 27.3% (validation) Epoch 18: 37.5% (train) vs. 28.6% (validation) Epoch 19: 40.3% (train) vs. 26.7% (validation) Epoch 20: 42.8% (train) vs. 26.6% (validation) Epoch 21: 46.2% (train) vs. 24.9% (validation) Epoch 22: 45.0% (train) vs. 29.5% (validation) Epoch 23: 50.3% (train) vs. 27.5% (validation) Epoch 24: 50.3% (train) vs. 28.0% (validation) Epoch 25: 50.7% (train) vs. 29.5% (validation) Epoch 26: 51.9% (train) vs. 25.0% (validation) Epoch 27: 55.8% (train) vs. 28.0% (validation) Epoch 28: 57.3% (train) vs. 25.8% (validation) Epoch 29: 59.4% (train) vs. 26.1% (validation) Epoch 30: 65.5% (train) vs. 24.0% (validation) Epoch 31: 67.0% (train) vs. 28.6% (validation) Epoch 32: 66.5% (train) vs. 26.7% (validation) Epoch 33: 68.9% (train) vs. 21.3% (validation) Epoch 34: 68.8% (train) vs. 23.0% (validation) Epoch 35: 70.5% (train) vs. 23.6% (validation) Epoch 36: 72.1% (train) vs. 25.3% (validation) Epoch 37: 70.6% (train) vs. 24.3% (validation) Epoch 38: 68.4% (train) vs. 25.0% (validation) Epoch 39: 67.1% (train) vs. 26.1% (validation) Epoch 40: 70.9% (train) vs. 24.4% (validation) Epoch 41: 73.2% (train) vs. 25.3% (validation) Epoch 42: 77.8% (train) vs. 24.3% (validation) Epoch 43: 79.7% (train) vs. 26.0% (validation) Epoch 44: 79.6% (train) vs. 26.4% (validation) Epoch 45: 78.3% (train) vs. 26.0% (validation) Epoch 46: 79.7% (train) vs. 26.6% (validation) Epoch 47: 80.7% (train) vs. 27.5% (validation) Epoch 48: 83.1% (train) vs. 26.3% (validation) Epoch 49: 80.6% (train) vs. 23.8% (validation) Epoch 50: 80.9% (train) vs. 23.8% (validation) Epoch 51: 81.9% (train) vs. 26.7% (validation) Epoch 52: 83.4% (train) vs. 24.9% (validation) Epoch 53: 85.0% (train) vs. 27.5% (validation) Epoch 54: 85.7% (train) vs. 24.4% (validation) Epoch 55: 84.7% (train) vs. 24.7% (validation) Epoch 56: 86.7% (train) vs. 25.3% (validation) Epoch 57: 87.6% (train) vs. 24.3% (validation) Epoch 58: 88.3% (train) vs. 22.9% (validation) Epoch 59: 86.4% (train) vs. 24.4% (validation) Epoch 60: 87.8% (train) vs. 26.4% (validation) Epoch 61: 87.6% (train) vs. 25.3% (validation) Epoch 62: 87.7% (train) vs. 26.9% (validation) Epoch 63: 89.1% (train) vs. 27.3% (validation) Epoch 64: 89.4% (train) vs. 27.6% (validation) Epoch 65: 89.1% (train) vs. 25.0% (validation) Epoch 66: 92.1% (train) vs. 28.0% (validation) Epoch 67: 88.8% (train) vs. 25.3% (validation) Epoch 68: 90.7% (train) vs. 25.3% (validation) Epoch 69: 91.1% (train) vs. 25.5% (validation) Epoch 70: 88.6% (train) vs. 27.5% (validation) Epoch 71: 88.8% (train) vs. 27.0% (validation) Epoch 72: 91.8% (train) vs. 25.8% (validation) Epoch 73: 92.8% (train) vs. 27.2% (validation) Epoch 74: 89.8% (train) vs. 24.4% (validation) Epoch 75: 90.1% (train) vs. 24.6% (validation) End: 74.8% (train) vs. 24.6% (validation) Going 0.0005 regularization Epoch 1: 7.2% (train) vs. 10.3% (validation) Epoch 2: 9.0% (train) vs. 7.6% (validation) Epoch 3: 7.0% (train) vs. 7.6% (validation) Epoch 4: 6.9% (train) vs. 7.6% (validation) Epoch 5: 6.7% (train) vs. 7.6% (validation) Epoch 6: 6.7% (train) vs. 7.6% (validation) Epoch 7: 6.7% (train) vs. 7.6% (validation) Epoch 8: 6.7% (train) vs. 7.6% (validation) Epoch 9: 6.5% (train) vs. 7.6% (validation) Epoch 10: 6.2% (train) vs. 7.6% (validation) Epoch 11: 6.2% (train) vs. 7.6% (validation) Epoch 12: 6.5% (train) vs. 7.6% (validation) Epoch 13: 6.6% (train) vs. 7.6% (validation) Epoch 14: 6.6% (train) vs. 7.6% (validation) Epoch 15: 6.7% (train) vs. 7.6% (validation) Epoch 16: 6.8% (train) vs. 7.6% (validation) Epoch 17: 6.9% (train) vs. 7.6% (validation) Epoch 18: 6.9% (train) vs. 7.6% (validation) Epoch 19: 6.9% (train) vs. 7.6% (validation) Epoch 20: 6.9% (train) vs. 7.6% (validation) Epoch 21: 6.9% (train) vs. 7.6% (validation) Epoch 22: 6.9% (train) vs. 7.6% (validation) Epoch 23: 6.9% (train) vs. 7.6% (validation) Epoch 24: 6.9% (train) vs. 7.6% (validation) Epoch 25: 7.0% (train) vs. 7.6% (validation) Epoch 26: 7.0% (train) vs. 7.6% (validation) Epoch 27: 7.0% (train) vs. 7.6% (validation) Epoch 28: 7.1% (train) vs. 7.6% (validation) Epoch 29: 7.1% (train) vs. 7.6% (validation) Epoch 30: 7.1% (train) vs. 7.6% (validation) Epoch 31: 7.1% (train) vs. 7.6% (validation) Epoch 32: 7.1% (train) vs. 7.6% (validation) Epoch 33: 7.0% (train) vs. 7.6% (validation) Epoch 34: 7.0% (train) vs. 7.6% (validation) Epoch 35: 7.0% (train) vs. 7.6% (validation) Epoch 36: 7.0% (train) vs. 7.6% (validation) Epoch 37: 7.0% (train) vs. 7.6% (validation) Epoch 38: 7.0% (train) vs. 7.6% (validation) Epoch 39: 7.0% (train) vs. 7.6% (validation) Epoch 40: 7.0% (train) vs. 7.6% (validation) Epoch 41: 7.0% (train) vs. 7.6% (validation) Epoch 42: 7.0% (train) vs. 7.6% (validation) Epoch 43: 7.0% (train) vs. 7.6% (validation) Epoch 44: 7.0% (train) vs. 7.6% (validation) Epoch 45: 7.0% (train) vs. 7.6% (validation) Epoch 46: 7.0% (train) vs. 7.6% (validation) Epoch 47: 7.0% (train) vs. 7.6% (validation) Epoch 48: 7.0% (train) vs. 7.6% (validation) Epoch 49: 7.0% (train) vs. 7.6% (validation) Epoch 50: 7.0% (train) vs. 7.6% (validation) Epoch 51: 7.0% (train) vs. 7.6% (validation) Epoch 52: 7.0% (train) vs. 7.6% (validation) Epoch 53: 7.0% (train) vs. 7.6% (validation) Epoch 54: 7.0% (train) vs. 7.6% (validation) Epoch 55: 7.0% (train) vs. 7.6% (validation) Epoch 56: 7.0% (train) vs. 7.6% (validation) Epoch 57: 7.0% (train) vs. 7.6% (validation) Epoch 58: 7.0% (train) vs. 7.6% (validation) Epoch 59: 7.0% (train) vs. 7.6% (validation) Epoch 60: 7.0% (train) vs. 7.6% (validation) Epoch 61: 7.0% (train) vs. 7.6% (validation) Epoch 62: 7.0% (train) vs. 7.6% (validation) Epoch 63: 7.0% (train) vs. 7.6% (validation) Epoch 64: 7.0% (train) vs. 7.6% (validation) Epoch 65: 7.0% (train) vs. 7.6% (validation) Epoch 66: 7.0% (train) vs. 7.6% (validation) Epoch 67: 7.0% (train) vs. 7.6% (validation) Epoch 68: 7.0% (train) vs. 7.6% (validation) Epoch 69: 7.0% (train) vs. 7.6% (validation) Epoch 70: 7.0% (train) vs. 7.6% (validation) Epoch 71: 7.0% (train) vs. 7.6% (validation) Epoch 72: 7.0% (train) vs. 7.6% (validation) Epoch 73: 7.0% (train) vs. 7.6% (validation) Epoch 74: 7.0% (train) vs. 7.6% (validation) Epoch 75: 7.0% (train) vs. 7.6% (validation) End: 7.6000000000000005% (train) vs. 7.6000000000000005% (validation)
with tf.Session() as sess:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0.0001]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_normal', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=50, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=25, input_dim=50, activation='relu', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=13, input_dim=25, activation='softmax', use_bias=True, kernel_initializer='random_normal', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1*0.1
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0001 regularization Epoch 1: 9.2% (train) vs. 7.6% (validation) Epoch 2: 7.5% (train) vs. 7.8% (validation) Epoch 3: 7.4% (train) vs. 11.9% (validation) Epoch 4: 8.4% (train) vs. 7.6% (validation) Epoch 5: 9.9% (train) vs. 12.6% (validation) Epoch 6: 11.9% (train) vs. 13.5% (validation) Epoch 7: 12.3% (train) vs. 13.6% (validation) Epoch 8: 14.2% (train) vs. 15.3% (validation) Epoch 9: 14.7% (train) vs. 17.2% (validation) Epoch 10: 16.9% (train) vs. 17.6% (validation) Epoch 11: 17.6% (train) vs. 18.9% (validation) Epoch 12: 18.9% (train) vs. 17.6% (validation) Epoch 13: 18.9% (train) vs. 18.9% (validation) Epoch 14: 20.6% (train) vs. 22.0% (validation) Epoch 15: 21.6% (train) vs. 21.5% (validation) Epoch 16: 23.8% (train) vs. 21.8% (validation) Epoch 17: 24.7% (train) vs. 22.4% (validation) Epoch 18: 24.4% (train) vs. 26.4% (validation) Epoch 19: 26.4% (train) vs. 24.3% (validation) Epoch 20: 28.6% (train) vs. 24.4% (validation) Epoch 21: 28.6% (train) vs. 24.0% (validation) Epoch 22: 29.8% (train) vs. 27.6% (validation) Epoch 23: 30.1% (train) vs. 25.3% (validation) Epoch 24: 30.6% (train) vs. 21.8% (validation) Epoch 25: 30.3% (train) vs. 22.4% (validation) Epoch 26: 32.0% (train) vs. 23.8% (validation) Epoch 27: 33.3% (train) vs. 24.6% (validation) Epoch 28: 34.1% (train) vs. 23.6% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-105-9dfd2d897842> in <module>() 55 cb = [MyCallback()] 56 ---> 57 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=16, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 58 59 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1*0.1 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
if True:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
#second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=80, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_uniform', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
#second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=40, input_dim=80, activation='relu', use_bias=True, kernel_initializer='random_uniform', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=13, input_dim=40, activation='softmax', use_bias=True, kernel_initializer='random_uniform', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1/10
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1/10
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 10.8% (train) vs. 9.8% (validation) Epoch 2: 12.9% (train) vs. 13.5% (validation) Epoch 3: 15.6% (train) vs. 16.4% (validation) Epoch 4: 17.1% (train) vs. 18.3% (validation) Epoch 5: 18.0% (train) vs. 18.9% (validation) Epoch 6: 19.5% (train) vs. 19.6% (validation) Epoch 7: 20.1% (train) vs. 20.0% (validation) Epoch 8: 21.2% (train) vs. 20.7% (validation) Epoch 9: 23.1% (train) vs. 21.0% (validation) Epoch 10: 23.6% (train) vs. 23.3% (validation) Epoch 11: 23.2% (train) vs. 25.0% (validation) Epoch 12: 26.7% (train) vs. 26.3% (validation) Epoch 13: 26.8% (train) vs. 26.4% (validation) Epoch 14: 27.9% (train) vs. 24.9% (validation) Epoch 15: 27.6% (train) vs. 24.9% (validation) Epoch 16: 30.5% (train) vs. 25.3% (validation) Epoch 17: 29.3% (train) vs. 27.3% (validation) Epoch 18: 31.3% (train) vs. 25.6% (validation) Epoch 19: 32.6% (train) vs. 25.3% (validation) Epoch 20: 35.1% (train) vs. 28.7% (validation) Epoch 21: 37.5% (train) vs. 26.0% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-143-0e3b30d50675> in <module>() 55 cb = [MyCallback()] 56 ---> 57 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 58 59 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1/10 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
'''class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 12479)
plt.imshow(shuffled_data[i][0])
l = second_iteration.predict(shuffled_data[i:1 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
print(f'Second option: {class_names[second]}')
print(f'Second confidence: {l[0][second]}')
print(f'Index: {i}')'''
class_names = ['sea animals', 'flowers', 'food containers', 'fruit and vegetables', 'electronics', 'furniture', 'insects', \
'land animals', 'invertebrates', 'people', 'reptiles', 'trees', 'vehicles']
from random import randint
i = randint(0, 649)
plt.imshow(shuffled_data[2600 + i][0])
l = second_iteration.predict(shuffled_data[2600 + i:2601 + i])
ind = [j for j in range(13) if l[0][j] == max(l[0])]
second = [j for j in range(13) if l[0][j]==sorted(l[0])[-2]][0]
print(f'Expected: {class_names[shuffled_y[2600 + i]]}')
print(f'Got: {class_names[ind[0]]}')
print(f'Confidence: {l[0][ind[0]]}')
if ind[0] != shuffled_y[2600 + i]: print(f'Confidence in the correct answer: {l[0][shuffled_y[2600 + i]]}')
print(f'Index: {i}')
Expected: flowers Got: electronics Confidence: 0.07768776267766953 Confidence in the correct answer: 0.07636738568544388 Index: 219
EDA, round 4.
Note: I know that today is supposed to be a "guiding deadline" to finish the product. But in light of the advanced nature of this project, I find it permissible to keep working on this until perhaps as late as next Wednesday night. The point is to get this thing to as high of a level as possible.
What I find is that the model struggles to cast out irrelevant details and focus on the "main object". For instance, if one shows the model an image of an orange placed on a table, clearly the orange will be the main focus of the image. But the model instead predicts "furniture", which isn't wrong but is just missing the point. In other cases, an image of a table will be presented without anything else, so clearly the focus of the image is on the table and the model should predict "furniture".
So overall the model struggles with focus. Perhaps this is why overfitting is perceived and also why it can't be fixed with traditional methods such as regularization. Addressing the overfitting issue requires addressing far more fundamental problems with the data. No samples need to be removed; the model just needs to learn focus. And many of the images in the dataset are weird in this regard. You could show it a fishbowl with a fish inside it and it would say "food containers" instead of "sea animals".
As for how I would address this issue, regularizing the convolutions would be my immediate thought. But it hasn't worked regardless of how many times I try it. This is something that I intend to discuss tomorrow with Mr. Errea during our second meeting.
if True:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow import keras
import tensorflow as tf
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["acc"]*100//0.1/10}% (train) vs. {logs["val_acc"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=32, strides=1, padding='same', kernel_initializer='random_uniform', bias_initializer='zeros', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dropout(0.1))
second_iteration.add(layers.Dense(units=200, input_dim=512, activation='relu', use_bias=True, kernel_initializer='random_uniform', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_initializer='random_uniform', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_initializer='random_uniform', bias_initializer='zeros', kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1/10
valid_acc = second_iteration.evaluate(shuffled_data[2600:], one_hot_encode(shuffled_y[2600:]), verbose=False)[1]*100//0.1/10
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Epoch 1: 6.8% (train) vs. 11.8% (validation) Epoch 2: 6.3% (train) vs. 7.6% (validation) Epoch 3: 6.6% (train) vs. 7.6% (validation) Epoch 4: 6.9% (train) vs. 7.6% (validation) Epoch 5: 7.2% (train) vs. 7.6% (validation)
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-246-3567a138b883> in <module>() 55 cb = [MyCallback()] 56 ---> 57 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb) 58 59 tr_acc = second_iteration.evaluate(shuffled_data[:2600], one_hot_encode(shuffled_y[:2600]), verbose=False)[1]*100//0.1/10 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
condensed_data = np.zeros((19500, 1, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((19500), dtype=np.uint8)
ptrs = [1500] * 13
training = 0
validation = 0
bluerept = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
blueback = False
for k in range(30):
for l in range(30):
cells = []
cells.append(classified_data_2[i][0][k][l])
cells.append(classified_data_2[i][0][k+1][l])
cells.append(classified_data_2[i][0][k+2][l])
cells.append(classified_data_2[i][0][k][l+1])
cells.append(classified_data_2[i][0][k+1][l+1])
cells.append(classified_data_2[i][0][k+2][l+1])
cells.append(classified_data_2[i][0][k][l+2])
cells.append(classified_data_2[i][0][k+1][l+2])
cells.append(classified_data_2[i][0][k+2][l+2])
cnt = 0
for cell in cells:
if (int(cell[2]) / 10 >= int(cell[0]) + int(cell[1])):
cnt += 1
if cnt == 9:
blueback = True
break
if blueback: break
if blueback and val == 10: continue
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: continue
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][0][k][l][0],classified_data_2[i][0][k][l][1],classified_data_2[i][0][k][l][2],\
classified_data_2[i][0][k+1][l][0],classified_data_2[i][0][k+1][l][1],classified_data_2[i][0][k+1][l][2],\
classified_data_2[i][0][k][l+1][0],classified_data_2[i][0][k][l+1][1],classified_data_2[i][0][k][l+1][2],\
classified_data_2[i][0][k+1][l+1][0],classified_data_2[i][0][k+1][l+1][1],classified_data_2[i][0][k+1][l+1][2])==255:
found = True
break
if found: break
if found: continue
if (training + validation) % 100 == 0: print(training + validation, end=" ")
if ptrs[val] <= 300:
condensed_data[15600 + validation] = classified_data_2[i]
condensed_y[15600 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((19500, 1, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((19500), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(15600)]
shuffled_valid = [*range(15600, 19500)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(15600):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(3900):
shuffled_data[shuffled_valid[i]] = condensed_data[15600 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[15600 + i]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500 15600 15700 15800 15900 16000 16100 16200 16300 16400 16500 16600 16700 16800 16900 17000 17100 17200 17300 17400 17500 17600 17700 17800 17900 18000 18100 18200 18300 18400 18500 18600 18700 18800 18900 19000 19100 19200 19300 19400
if True:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow.keras import initializers
from tensorflow import keras
import tensorflow as tf
from math import sqrt
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["accuracy"]*100//0.1/10}% (train) vs. {logs["val_accuracy"]*100//0.1/10}% (validation)')
for val in [0]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(1, 32, 32, 3)))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/3072))))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/16384))))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/16384))))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/4096))))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/4096))))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/4096))))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/1024))))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/1024))))
second_iteration.add(layers.Conv3D(activation='relu', kernel_size=3, filters=16, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/1024))))
second_iteration.add(layers.MaxPool3D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.Dense(units=60, input_dim=256, activation='relu', use_bias=True, kernel_regularizer=regularizers.l2(0), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/256))))
#second_iteration.add(layers.Dense(units=30, input_dim=60, activation='relu', use_bias=True, kernel_regularizer=regularizers.l2(0)))
second_iteration.add(layers.Dense(units=13, input_dim=60, activation='softmax', use_bias=True, kernel_regularizer=regularizers.l2(0), kernel_initializer=initializers.RandomNormal(mean=0.0, stddev=sqrt(2/60))))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = []
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=True, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:15600], one_hot_encode(shuffled_y[:15600]), verbose=False)[1]*100//0.1/10
valid_acc = second_iteration.evaluate(shuffled_data[15600:], one_hot_encode(shuffled_y[15600:]), verbose=False)[1]*100//0.1/10
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0 regularization Train on 15600 samples, validate on 3900 samples Epoch 1/75 15600/15600 [==============================] - 244s 16ms/step - loss: 2.4602 - acc: 0.1291 - val_loss: 2.3589 - val_acc: 0.1836 Epoch 2/75 15600/15600 [==============================] - 177s 11ms/step - loss: 2.2655 - acc: 0.2295 - val_loss: 2.2794 - val_acc: 0.2238 Epoch 3/75 12704/15600 [=======================>......] - ETA: 30s - loss: 2.1699 - acc: 0.2658
--------------------------------------------------------------------------- KeyboardInterrupt Traceback (most recent call last) <ipython-input-253-a87c11379f20> in <module>() 60 cb = [] 61 ---> 62 second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=75, validation_split=0.2, shuffle=False, verbose=True, callbacks=cb) 63 64 tr_acc = second_iteration.evaluate(shuffled_data[:15600], one_hot_encode(shuffled_y[:15600]), verbose=False)[1]*100//0.1/10 C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, max_queue_size, workers, use_multiprocessing, **kwargs) 1637 initial_epoch=initial_epoch, 1638 steps_per_epoch=steps_per_epoch, -> 1639 validation_steps=validation_steps) 1640 1641 def evaluate(self, C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\engine\training_arrays.py in fit_loop(model, inputs, targets, sample_weights, batch_size, epochs, verbose, callbacks, val_inputs, val_targets, val_sample_weights, shuffle, initial_epoch, steps_per_epoch, validation_steps) 213 ins_batch[i] = ins_batch[i].toarray() 214 --> 215 outs = f(ins_batch) 216 if not isinstance(outs, list): 217 outs = [outs] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\keras\backend.py in __call__(self, inputs) 2984 2985 fetched = self._callable_fn(*array_vals, -> 2986 run_metadata=self.run_metadata) 2987 self._call_fetch_callbacks(fetched[-len(self._fetches):]) 2988 return fetched[:len(self.outputs)] C:\Users\jo08t\anaconda3\envs\exts-ml\lib\site-packages\tensorflow\python\client\session.py in __call__(self, *args, **kwargs) 1437 ret = tf_session.TF_SessionRunCallable( 1438 self._session._session, self._handle, args, status, -> 1439 run_metadata_ptr) 1440 if run_metadata: 1441 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr) KeyboardInterrupt:
condensed_data = np.zeros((19500, 32, 32, 3), dtype=np.uint8)
condensed_y = np.zeros((19500), dtype=np.uint8)
ptrs = [1500] * 13
training = 0
validation = 0
bluerept = 0
for i in range(37500):
if max(ptrs) == 0: break
val = int(y_data[i])
if val in [7, 8, 9]: val = 7
elif val in [10, 11, 12, 13, 14]: val -= 2
if ptrs[val] == 0: continue
blueback = False
for k in range(30):
for l in range(30):
cells = []
cells.append(classified_data_2[i][0][k][l])
cells.append(classified_data_2[i][0][k+1][l])
cells.append(classified_data_2[i][0][k+2][l])
cells.append(classified_data_2[i][0][k][l+1])
cells.append(classified_data_2[i][0][k+1][l+1])
cells.append(classified_data_2[i][0][k+2][l+1])
cells.append(classified_data_2[i][0][k][l+2])
cells.append(classified_data_2[i][0][k+1][l+2])
cells.append(classified_data_2[i][0][k+2][l+2])
cnt = 0
for cell in cells:
if (int(cell[2]) / 10 >= int(cell[0]) + int(cell[1])):
cnt += 1
if cnt == 9:
blueback = True
break
if blueback: break
if blueback and val == 10: continue
is_grayscale = True
for j in range(32):
for k in range(32):
one = classified_data_2[i][0][j][k][0]
two = classified_data_2[i][0][j][k][1]
three = classified_data_2[i][0][j][k][2]
if max(one, two, three) != min(one, two, three):
is_grayscale = False
break
if not is_grayscale:
break
if is_grayscale: continue
found = False
for k in range(31):
for l in range(31):
if min(classified_data_2[i][0][k][l][0],classified_data_2[i][0][k][l][1],classified_data_2[i][0][k][l][2],\
classified_data_2[i][0][k+1][l][0],classified_data_2[i][0][k+1][l][1],classified_data_2[i][0][k+1][l][2],\
classified_data_2[i][0][k][l+1][0],classified_data_2[i][0][k][l+1][1],classified_data_2[i][0][k][l+1][2],\
classified_data_2[i][0][k+1][l+1][0],classified_data_2[i][0][k+1][l+1][1],classified_data_2[i][0][k+1][l+1][2])==255:
found = True
break
if found: break
if found: continue
if (training + validation) % 100 == 0: print(training + validation, end=" ")
if ptrs[val] <= 300:
condensed_data[15600 + validation] = classified_data_2[i][0]
condensed_y[15600 + validation] = val
validation += 1
else:
condensed_data[training] = classified_data_2[i][0]
condensed_y[training] = val
training += 1
ptrs[val] -= 1
shuffled_data = np.zeros((19500, 32, 32, 3), dtype=np.uint8)
shuffled_y = np.zeros((19500), dtype=np.uint8)
from random import shuffle
shuffled_train = [*range(15600)]
shuffled_valid = [*range(15600, 19500)]
shuffle(shuffled_train)
shuffle(shuffled_valid)
for i in range(15600):
shuffled_data[shuffled_train[i]] = condensed_data[i]
shuffled_y[shuffled_train[i]] = condensed_y[i]
for i in range(3900):
shuffled_data[shuffled_valid[i]] = condensed_data[15600 + i]
shuffled_y[shuffled_valid[i]] = condensed_y[15600 + i]
0 100 200 300 400 500 600 700 800 900 1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000 2100 2200 2300 2400 2500 2600 2700 2800 2900 3000 3100 3200 3300 3400 3500 3600 3700 3800 3900 4000 4100 4200 4300 4400 4500 4600 4700 4800 4900 5000 5100 5200 5300 5400 5500 5600 5700 5800 5900 6000 6100 6200 6300 6400 6500 6600 6700 6800 6900 7000 7100 7200 7300 7400 7500 7600 7700 7800 7900 8000 8100 8200 8300 8400 8500 8600 8700 8800 8900 9000 9100 9200 9300 9400 9500 9600 9700 9800 9900 10000 10100 10200 10300 10400 10500 10600 10700 10800 10900 11000 11100 11200 11300 11400 11500 11600 11700 11800 11900 12000 12100 12200 12300 12400 12500 12600 12700 12800 12900 13000 13100 13200 13300 13400 13500 13600 13700 13800 13900 14000 14100 14200 14300 14400 14500 14600 14700 14800 14900 15000 15100 15200 15300 15400 15500 15600 15700 15800 15900 16000 16100 16200 16300 16400 16500 16600 16700 16800 16900 17000 17100 17200 17300 17400 17500 17600 17700 17800 17900 18000 18100 18200 18300 18400 18500 18600 18700 18800 18900 19000 19100 19200 19300 19400
if True:
def one_hot_encode(arr):
res=np.zeros((arr.size, 13), dtype=np.uint8)
for i in range(arr.size):
res[i][int(arr[i])]=1
return res
from tensorflow.keras import regularizers
from tensorflow.keras import losses
from tensorflow.keras import callbacks
from tensorflow.keras import models, layers
from tensorflow.keras import initializers
from tensorflow import keras
import tensorflow as tf
from math import sqrt
def scaled_loss(y_true, y_pred):
cce = losses.categorical_crossentropy(y_true, y_pred)
chosen = tf.math.argmax(y_true)
if str(chosen.shape[0]) != '?' and (int(chosen.eval()[0]), int(y_pred.eval()[0])) in ((2, 3), (3, 2), (4, 5), (5, 4), (6, 8), (6, 11), (0, 10), (2, 4), (2, 5), (4, 2), (5, 2)):
cce = tf.multiply(cce, 10000)
return cce
class MyCallback(callbacks.Callback):
def on_epoch_end(self, epoch, logs=None):
print(f'Epoch {epoch + 1}: {logs["accuracy"]*100//0.1/10}% (train) vs. {logs["val_accuracy"]*100//0.1/10}% (validation)')
for val in [5e-4]:
print(f'Going {val} regularization')
second_iteration = models.Sequential()
second_iteration.add(layers.InputLayer(input_shape=(32, 32, 3)))
second_iteration.add(layers.RandomCrop(25, 25))
second_iteration.add(layers.RandomZoom((0, 0.4)))
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool2D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool2D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.MaxPool2D(pool_size=2, strides=2, padding='same'))
second_iteration.add(layers.Flatten())
second_iteration.add(layers.BatchNormalization())
#second_iteration.add(layers.Dropout(0.05))
second_iteration.add(layers.Dense(units=200, input_dim=1024, activation='relu', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=100, input_dim=200, activation='relu', use_bias=True, kernel_regularizer=regularizers.l2(val)))
second_iteration.add(layers.BatchNormalization())
second_iteration.add(layers.Dropout(0.2))
second_iteration.add(layers.Dense(units=13, input_dim=100, activation='softmax', use_bias=True, kernel_regularizer=regularizers.l2(val)))
opt = keras.optimizers.Adam()
second_iteration.compile(opt, 'categorical_crossentropy', ['accuracy'])
cb = [MyCallback()]
second_iteration.fit(shuffled_data, one_hot_encode(shuffled_y), batch_size=32, epochs=5000, validation_split=0.2, shuffle=False, verbose=False, callbacks=cb)
tr_acc = second_iteration.evaluate(shuffled_data[:15600], one_hot_encode(shuffled_y[:15600]), verbose=False)[1]*100//0.1/10
valid_acc = second_iteration.evaluate(shuffled_data[15600:], one_hot_encode(shuffled_y[15600:]), verbose=False)[1]*100//0.1/10
print(f'End: {tr_acc}% (train) vs. {valid_acc}% (validation)')
Going 0.0005 regularization
--------------------------------------------------------------------------- AttributeError Traceback (most recent call last) <ipython-input-255-6692e208e955> in <module>() 34 35 second_iteration.add(layers.InputLayer(input_shape=(32, 32, 3))) ---> 36 second_iteration.add(layers.RandomCrop(25, 25)) 37 second_iteration.add(layers.RandomZoom((0, 0.4))) 38 second_iteration.add(layers.Conv2D(activation='relu', kernel_size=3, filters=64, strides=1, padding='same', use_bias=True, kernel_regularizer=regularizers.l2(val))) AttributeError: module 'tensorflow._api.v1.keras.layers' has no attribute 'RandomCrop'
dir(layers)
['Activation', 'ActivityRegularization', 'Add', 'AlphaDropout', 'Average', 'AveragePooling1D', 'AveragePooling2D', 'AveragePooling3D', 'AvgPool1D', 'AvgPool2D', 'AvgPool3D', 'BatchNormalization', 'Bidirectional', 'Concatenate', 'Conv1D', 'Conv2D', 'Conv2DTranspose', 'Conv3D', 'Conv3DTranspose', 'ConvLSTM2D', 'Convolution1D', 'Convolution2D', 'Convolution2DTranspose', 'Convolution3D', 'Convolution3DTranspose', 'Cropping1D', 'Cropping2D', 'Cropping3D', 'CuDNNGRU', 'CuDNNLSTM', 'Dense', 'DepthwiseConv2D', 'Dot', 'Dropout', 'ELU', 'Embedding', 'Flatten', 'GRU', 'GRUCell', 'GaussianDropout', 'GaussianNoise', 'GlobalAveragePooling1D', 'GlobalAveragePooling2D', 'GlobalAveragePooling3D', 'GlobalAvgPool1D', 'GlobalAvgPool2D', 'GlobalAvgPool3D', 'GlobalMaxPool1D', 'GlobalMaxPool2D', 'GlobalMaxPool3D', 'GlobalMaxPooling1D', 'GlobalMaxPooling2D', 'GlobalMaxPooling3D', 'Input', 'InputLayer', 'InputSpec', 'LSTM', 'LSTMCell', 'Lambda', 'Layer', 'LeakyReLU', 'LocallyConnected1D', 'LocallyConnected2D', 'Masking', 'MaxPool1D', 'MaxPool2D', 'MaxPool3D', 'MaxPooling1D', 'MaxPooling2D', 'MaxPooling3D', 'Maximum', 'Minimum', 'Multiply', 'PReLU', 'Permute', 'RNN', 'ReLU', 'RepeatVector', 'Reshape', 'SeparableConv1D', 'SeparableConv2D', 'SeparableConvolution1D', 'SeparableConvolution2D', 'SimpleRNN', 'SimpleRNNCell', 'Softmax', 'SpatialDropout1D', 'SpatialDropout2D', 'SpatialDropout3D', 'StackedRNNCells', 'Subtract', 'ThresholdedReLU', 'TimeDistributed', 'UpSampling1D', 'UpSampling2D', 'UpSampling3D', 'Wrapper', 'ZeroPadding1D', 'ZeroPadding2D', 'ZeroPadding3D', '__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__path__', '__spec__', 'add', 'average', 'concatenate', 'dot', 'maximum', 'minimum', 'multiply', 'subtract']